Speech synthesis method, speech synthesis device, electronic device, and storage medium

Abstract

The speech synthesis method, the speech synthesis device, the electronic equipment and the storage medium provided by the embodiment of the present application relate to the technical field of financial technology. The method comprises the following steps: performing spectrum feature extraction on original speech data to obtain original speech mel-frequency cepstrum and original speech linear spectrum; performing feature coding on the original speech mel-frequency cepstrum through a preset speaker encoder to obtain original speaker features; performing posterior coding on the original speech linear spectrum through a preset posterior encoder to obtain speech posterior features; performing content feature extraction according to the original speaker features and the speech posterior features to obtain target speech content features; and performing speech synthesis according to preset target speaker features and the target speech content features to obtain synthesized speech data; wherein the synthesized speech data is used to represent speech uttered by a target speaker object according to original speech content. The embodiment of the present application can guarantee the quality of speech synthesis and can perform speech conversion for any speaker.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of financial technology, and in particular to a speech synthesis method, a speech synthesis device, an electronic device and a storage medium. BACKGROUND

[0002] The purpose of speech conversion in speech synthesis is to change the voice of the source speaker so that it sounds like the target speaker. For example, in banks, insurance companies or other financial institutions, intelligent customer service often guides users to handle corresponding businesses through voice. When guiding users through voice, in order to improve the guiding effect or based on user customization needs, it is often necessary to replace the source speaker in the voice with the target speaker. At present, with the rapid development of acoustic models and high-quality neural network vocoders, speaker conversion in speech synthesis has made significant progress. Under the condition of a large amount of high-quality recording data of speakers, existing speech synthesis models can easily realize speech conversion between multiple speakers. However, it is difficult to obtain high-quality recording data of new speakers, which makes it difficult to realize speech conversion customization services for new speakers. Therefore, how to provide a speech synthesis method that can guarantee the quality of speech synthesis and can perform speech conversion for any speaker has become a technical problem to be solved. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a speech synthesis method, a speech synthesis device, an electronic device and a storage medium, which can guarantee the quality of speech synthesis and can perform speech conversion for any speaker.

[0004] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application provides a speech synthesis method, which comprises:

[0005] Obtaining original speech data; the original speech data is used to represent the speech uttered by an original speaker object according to a preset original speech content;

[0006] Performing spectral feature extraction on the original speech data to obtain original speech mel-frequency cepstrum and original speech linear spectrum;

[0007] Encoding the original speech mel-frequency cepstrum through a preset speaker encoder to obtain original speaker features; wherein the original speaker features are used to represent the speaking features of the original speaker object;

[0008] Performing posterior encoding on the original speech linear spectrum through a preset posterior encoder to obtain speech posterior features; the speech posterior features are used to represent the speaking features of the original speaker object and the speech content features of the original speech content;

[0009] extracting a content feature according to the original speaker feature and the speech posterior feature, to obtain a target speech content feature;

[0010] performing speech synthesis according to a preset target speaker feature and the target speech content feature, to obtain synthesized speech data; wherein the target speaker feature is used to represent a speaking feature of a target speaker object, and the synthesized speech data is used to represent a speech uttered by the target speaker object according to the original speech content.

[0011] In some embodiments, the extracting a content feature according to the original speaker feature and the speech posterior feature, to obtain a target speech content feature, comprises:

[0012] extracting a feature of the speech posterior feature through a preset content feature extraction model, to obtain a mapping content feature;

[0013] adjusting the mapping content feature according to the original speaker feature, to obtain the target content feature.

[0014] In some embodiments, before the extracting a feature of the speech posterior feature through a preset content feature extraction model, to obtain a mapping content feature, the method further comprises:

[0015] training the content feature extraction model, specifically comprising:

[0016] extracting a feature of the speech posterior feature through a preset original feature extraction model, to obtain an initial speech content feature;

[0017] performing speech synthesis according to the original speaker feature and the initial speech content feature, to obtain intermediate speech data;

[0018] extracting a spectral feature of the intermediate speech data, to obtain an intermediate speech linear spectrum;

[0019] calculating a loss according to the intermediate speech linear spectrum and the original speech linear spectrum, to obtain first loss data;

[0020] adjusting a parameter of the original feature extraction model according to the first loss data, to obtain the content feature extraction model.

[0021] In some embodiments, the speaker encoder comprises a first convolutional layer and a nonlinear neural network, and the extracting a feature of the original speech mel-frequency cepstrum through a preset speaker encoder, to obtain an original speaker feature, comprises:

[0022] extracting a hidden state feature of the original speech mel-frequency cepstrum through the first convolutional layer, to obtain a first hidden state speaker feature;

[0023] performing time length processing on the first hidden state speaker feature through the nonlinear neural network to obtain the original speaker feature.

[0024] In some embodiments, the nonlinear neural network comprises a residual layer, a second convolutional layer and a nonlinear subnetwork, and the performing time length processing on the first hidden state speaker feature through the nonlinear neural network to obtain the original speaker feature comprises:

[0025] performing hidden state feature activation on the first hidden state speaker feature through the residual layer to obtain a target activated speaker feature;

[0026] performing hidden state feature extraction on the target activated speaker feature through the second convolutional layer to obtain a second hidden state speaker feature;

[0027] performing time length processing on the second hidden state speaker feature through the nonlinear subnetwork to obtain the original speaker feature.

[0028] In some embodiments, the residual layer comprises a first convolutional kernel, a second convolutional kernel and a target activation function, and the performing hidden state feature activation on the first hidden state speaker feature through the residual layer to obtain a target activated speaker feature comprises:

[0029] performing feature extraction on the first hidden state speaker feature through the first convolutional kernel to obtain a first intermediate hidden state feature;

[0030] performing nonlinear transformation processing on the first hidden state speaker feature through the target activation function to obtain a target nonlinear hidden state feature;

[0031] performing feature extraction on the target nonlinear hidden state feature through the second convolutional kernel to obtain a second intermediate hidden state feature;

[0032] performing feature fusion on the first intermediate hidden state feature and the second intermediate hidden state feature to obtain the target activated speaker feature.

[0033] In some embodiments, the target activation function comprises a first activation function and a second activation function, and the performing nonlinear transformation processing on the first hidden state feature through the target activation function to obtain a target nonlinear hidden state feature comprises:

[0034] performing nonlinear transformation processing on the first hidden state feature through the first activation function to obtain a first nonlinear hidden state feature;

[0035] performing nonlinear transformation processing on the first hidden state feature through the second activation function to obtain a second nonlinear hidden state feature;

[0036] point-multiplying the first nonlinear hidden state feature and the second nonlinear hidden state feature to obtain the target nonlinear hidden state feature.

[0037] To achieve the above object, a second aspect of the embodiment of the present application provides a speech synthesis device, which comprises:

[0038] a speech data acquisition module configured to acquire original speech data, wherein the original speech data is used to represent speech uttered by an original speaker according to preset original speech content;

[0039] a spectrum extraction module configured to perform spectrum feature extraction on the original speech data to obtain original speech mel-cepstrum and original speech linear spectrum;

[0040] a speaker feature encoding module configured to perform feature encoding on the original speech mel-cepstrum by using a preset speaker encoder to obtain original speaker features, wherein the original speaker features are used to represent speaking features of the original speaker;

[0041] a posterior feature encoding module configured to perform posterior encoding on the original speech linear spectrum by using a preset posterior encoder to obtain speech posterior features, wherein the speech posterior features are used to represent speaking features of the original speaker and speech content features of the original speech content;

[0042] a content feature extraction module configured to perform content feature extraction according to the original speaker features and the speech posterior features to obtain target speech content features;

[0043] a speech synthesis module configured to perform speech synthesis according to preset target speaker features and the target speech content features to obtain synthesized speech data, wherein the target speaker features are used to represent speaking features of a target speaker, and the synthesized speech data is used to represent speech uttered by the target speaker according to the original speech content.

[0044] To achieve the above object, a third aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the speech synthesis method of the first aspect when executing the computer program.

[0045] To achieve the above object, a fourth aspect of the embodiment of the present application provides a storage medium, which is a computer readable storage medium, and stores a computer program, wherein the computer program is executed by a processor to implement the speech synthesis method of the first aspect.

[0046] The voice synthesis method, voice synthesis device, electronic equipment and storage medium provided by the present application, the method comprises: performing spectral feature extraction on original voice data to obtain original voice mel-frequency cepstrum and original voice linear spectrum. The original speaker feature can be obtained by performing feature coding on the original voice mel-frequency cepstrum through a speaker encoder. The voice posterior feature can be obtained by performing feature coding on the original voice linear spectrum through a posterior feature encoder. The target voice content feature irrelevant to the speaker feature can be obtained by performing content feature extraction on the original speaker feature and the voice posterior feature. Finally, the synthesized voice data is obtained by performing voice synthesis according to the target speaker feature and the target voice content feature. In summary, the embodiment of the present application can realize voice synthesis and voice conversion for any speaker. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a system architecture schematic diagram provided by the embodiment of the present application for performing the voice synthesis method;

[0048] Figure 2 is a flowchart of the voice synthesis method provided by the embodiment of the present application;

[0049] Figure 3 is Figure 2 the flowchart of step S103 in

[0050] Figure 4 is Figure 3 the flowchart of step S202 in

[0051] Figure 5 is Figure 4 the flowchart of step S301 in

[0052] Figure 6 is Figure 2 the flowchart of step S105 in

[0053] Figure 7 is a flowchart of the voice synthesis method provided by another embodiment of the present application;

[0054] Figure 8 is a module structure block diagram of the voice synthesis device provided by the embodiment of the present application;

[0055] Figure 9 is a hardware structure schematic diagram of the electronic equipment provided by the embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0057] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical sequence is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the device or the sequence in the flowchart. The terms "first", "second", and the like in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification herein is for the purpose of describing the embodiments of the present application only and is not intended to be limiting of the present application.

[0059] First, the meanings of several terms involved in the present application are explained:

[0060] Artificial intelligence (AI): is a new technical science that studies, develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, and artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, to perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0061] Natural language processing (NLP): NLP uses computers to process, understand and use human language (such as Chinese, English, etc.), and NLP is a branch of artificial intelligence, which is an interdisciplinary subject of computer science and linguistics, and is also commonly known as computational linguistics. Natural language processing includes syntax analysis, semantic analysis, discourse understanding, etc. Natural language processing is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information image processing, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining, etc. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and language computing related linguistic research.

[0062] Voice conversion technology: refers to a technology of converting original voice data into target voice data while keeping the semantic content unchanged, wherein the original voice data is the voice of an original speaker, and the target voice data is the voice of a target speaker, that is, the original voice of the original speaker is converted into the target voice of the target speaker with the same semantic content through the voice conversion technology.

[0063] In recent years, with the rapid development of sequence-to-sequence acoustic models and high-quality neural network vocoders, voice conversion has made significant progress. Under the condition of a large number of high-quality recording data of speakers, these models can easily realize voice conversion between multiple speakers. However, it is difficult to obtain high-quality recording data of new speakers, which makes it difficult to quickly realize voice conversion customized services for new speakers. The new speaker adaptation problem in voice conversion can be divided into few-shot and zero-shot according to the amount of data. Few-shot uses a small amount of data of a new speaker to realize voice conversion, usually by fine-tuning the overall or part of the parameters of a pre-trained model. Therefore, at least a small amount of voice data of a new speaker is needed to participate in the training of the model in few-shot voice conversion. However, the voice data of the new speaker is difficult to obtain and is usually insufficient to adjust the model, resulting in low accuracy of voice conversion and thus unable to realize voice conversion for any speaker.

[0064] On the contrary to few-shot, zero-shot tends to use only one piece of data of a new speaker to realize voice conversion, and does not need model fine-tuning. The embodiments of the present application provide a voice synthesis method for the zero-shot voice conversion problem, aiming to ensure the quality of voice synthesis while being able to realize voice conversion for any speaker.

[0065] In the field of financial technology, intelligent customer service voice guidance technology is becoming more and more common. For example, in XX bank, a voice database stores various guidance voice data in advance, and different guidance voice data corresponds to different speaking objects. Then a target guidance voice data of intelligent customer service can be selected from the voice database according to the demand, so as to realize voice guidance of different speaking objects. However, if the voice database does not store the guidance voice data of the target speaking object in advance, the voice guidance of the target speaking object cannot be realized. With the development of voice conversion technology, the speaker of the original voice data can be converted, that is, the original speaking object in the guidance voice data is converted into the target speaking object, and the original voice content remains unchanged. For example, a voice conversion model is generally used to convert the speaker of the original voice data to obtain target voice data, which is emitted by the target speaking object. However, the voice conversion model needs to be trained according to the voice data of the target speaking object to ensure the accuracy of the conversion, which greatly affects the efficiency of voice conversion. In addition, for a new speaking object, if a large amount of voice data of the new speaking object is not obtained and used for model training or model fine-tuning, the existing voice conversion model cannot realize voice conversion for the new speaking object. Therefore, the embodiment of the present application provides a voice synthesis method. When the voice synthesis method of the embodiment of the present application is applied to intelligent customer service voice guidance in the field of financial technology, and assuming that an object is a target speaking object, the target speaker feature can be extracted from a small amount of voice data of the target speaking object to realize voice conversion. Without obtaining a large amount of voice data of the target speaking object and without using the voice data for model training or model fine-tuning, the efficiency of voice conversion is improved. The specific processing details and other effects of the embodiment of the present application will be described in detail below.

[0066] The voice synthesis method provided by the embodiment of the present application can be applied to a server end, and can also be software running in the server end. The server end can be configured as a separate physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms; the software can be an application that implements the voice synthesis method, but is not limited to the above forms.

[0067] The application is operable in a variety of general purpose or special purpose computer systems environments or configurations. Examples of well known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computer systems, server computer systems, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PC systems, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0068] The embodiments of the application provide a speech synthesis method and a speech synthesis device, an electronic device and a storage medium. The embodiments are described as follows. First, a speech synthesis method in the embodiments of the application is described.

[0069] With reference to Figure 1 The speech synthesis method in the embodiments of the application can be executed by the target server 400 alone, executed by the first terminal 100, the second terminal 200 or the third terminal 300 alone, or executed by the first terminal 100, the second terminal 200, the third terminal 300 and the target server 400 together.

[0070] It should be noted that in each of the specific embodiments of the application, when it is necessary to perform relevant processing according to the speech data of the user and other data related to the identity or characteristics of the user, the permission or consent of the user is obtained first, and the collection, use and processing of the data comply with relevant laws, regulations and standards.

[0071] Figure 2 The speech synthesis method provided by the embodiments of the application is an optional flowchart, which can include but is not limited to steps S101 to S106.

[0072] In step S101, original speech data is obtained. The original speech data is used to represent speech uttered by an original speaker according to preset original speech content.

[0073] In step S102, spectrum feature extraction is performed on the original speech data to obtain original speech mel-cepstrum and original speech linear spectrum.

[0074] In step S103, feature coding is performed on the original speech mel-cepstrum by using a preset speaker encoder to obtain original speaker features. The original speaker features are used to represent the speaking features of the original speaker.

[0075] Step S104, performing post-processing coding on the original speech linear spectrum through a preset post-processing coder to obtain speech post-processing features; the speech post-processing features are used to represent the speaking features of the original speaker and the speech content features of the original speech content;

[0076] Step S105, performing content feature extraction according to the original speaker features and the speech post-processing features to obtain target speech content features;

[0077] Step S106, performing speech synthesis according to the preset target speaker features and the target speech content features to obtain synthesized speech data; wherein the target speaker features are used to represent the speaking features of the target speaker, and the synthesized speech data is used to represent the speech uttered by the target speaker according to the original speech content.

[0078] The steps S101 to S106 shown in the embodiments of the present application are used to perform frequency spectrum feature extraction on the original speech data to obtain original speech mel-frequency cepstrum and original speech linear spectrum. The original speech mel-frequency cepstrum is used to represent the speaking feature information of the original speaker, and the original speech linear spectrum is used to represent the speaking feature information of the original speaker and the speech feature information of the original speech content. Therefore, the original speaker features can be obtained by performing feature coding on the original speech mel-frequency cepstrum through the speaker coder, and the speech post-processing features can be obtained by performing feature coding on the original speech linear spectrum through the post-processing feature coder. The target speech content features can be obtained by performing content feature extraction on the original speaker features and the speech post-processing features, which is equivalent to removing the features related to the original speaker features from the speech post-processing features. Finally, the synthesized speech data can be obtained by performing speech synthesis according to the target speaker features and the target speech content features. In summary, the embodiments of the present application can realize speech synthesis and speech conversion for any speaker. In addition, the embodiments of the present application do not need to use the speech data of the target speaker to participate in model fine-tuning, and the efficiency of speech conversion is also improved.

[0079] For example, the speaker A utters the speech data M as "Hello, welcome to ×× bank, I am an intelligent customer service, please directly say the business you want to handle, please press 1 for business inquiry, please press 2 for business handling, and please press 3 for manual service", which has good guiding effect on the user and good user experience. The speech data M can be used as the original speech data in step S101. Then the speaker features of the speaker B are used as the target speaker features in step S106. Finally, the synthesized speech data is obtained according to the speech data M and the speaker features of the speaker B. The synthesized speech data is uttered by the speaker B, but the content and emotion in the speech data M are retained, so that the synthesized speech data also has the guiding effect on the user as the speech data M, and the flexibility of the speaker in the speech data is also improved.

[0080] It should be noted that the speaker feature of the speaker B is extracted from the speech data of the speaker B. However, the speech data of the speaker B is relatively small in the database (for example, due to legal restrictions, the speech data of the speaker B cannot be collected in large quantities), so that the small amount of speech data is low in speech conversion accuracy even if it participates in the training of the speech conversion model in the related technology. The embodiment of the present application does not need to use the speech data of the speaker B to participate in the model training, but only extracts the speaker feature from the small amount of speech data of the speaker B. Therefore, the embodiment of the present application also overcomes the technical problem of low speech conversion accuracy due to difficulty in collecting speech data. Especially when the embodiment is applied to the field of financial technology, the efficiency and flexibility of speech conversion are greatly improved, and compared with the related technology, it is more universal.

[0081] The above is a general description of steps S101 to S106, and the steps S101 to S106 will be described in detail below.

[0082] In step S101 of some embodiments, the original speech data is obtained; the original speech data is used to represent the speech uttered by the original speaker object according to the preset original speech content.

[0083] Specifically, when the speech synthesis method is applied in the first terminal 100, the original speech data can be obtained by recording, Bluetooth transmission, wired transmission, or downloading, and when the original speech data is obtained by recording, the first terminal 100 is correspondingly configured with a microphone and performs audio acquisition through the microphone to realize recording of the original speech data. When the speech synthesis method is applied in the target server 400, the original speech data can be uploaded to the target server 400 by the first terminal 100, or downloaded from other servers by the target server 400.

[0084] When the speaker object is distinguished, it can be distinguished by timbre. Because the vocal cords and the vibration mode of the vocal cords of different people are different, the vibration frequency, vibration amplitude and other parameters of the sound are also different, so that the timbres of different speaker objects are different. Therefore, the speaker feature of the embodiment of the present application can also be called timbre feature.

[0085] The original speech content is used to represent the text information of the original speech data. It should be noted that the original speech data can be a Chinese composed voice, or an English composed voice, etc.

[0086] In step S102 of some embodiments, the original speech data is subjected to spectrum feature extraction to obtain original speech mel-cepstrum and original speech linear spectrum.

[0087] Specifically, the spectral feature extraction can be performed on the original speech data by using an existing speech analysis algorithm, including a Speech Transformation and Representation using Adaptive Interpolation of weighted spectrum (STRAIGHT) algorithm and a discrete Fourier transform algorithm. For example, short-time Fourier transform is performed on the original speech data to obtain original speech linear spectrum; and mel filtering is performed on the original speech linear spectrum according to a preset mel-frequency cepstrum filter bank to obtain original speech mel-frequency cepstrum. The original speech mel-frequency cepstrum is calculated by using short-time Fourier transform to calculate the amplitude spectrum of the original speech data, and the original speech linear spectrum. The specific parameters of the short-time Fourier transform can be: a frame length of 20 ms, a frame shift of 10 ms, and a number of FFT points of 512. Then, the original speech linear spectrum is filtered by using a 64-dimensional mel-frequency cepstrum filter bank to obtain the original speech mel-frequency cepstrum. The feature dimension of the original speech mel-frequency cepstrum is T*64, where T is the number of frames of the original speech data.

[0088] In an embodiment, before step S103, the speech synthesis method of the embodiment further includes:

[0089] The speaker encoder or the posterior encoder is trained, specifically including: obtaining sample speech data; the sample speech data is used to represent speech uttered by a first speaker according to preset sample speech content; performing spectral feature extraction on the sample speech data to obtain sample original speech mel-frequency cepstrum and sample original speech linear spectrum; performing feature encoding on the sample speech mel-frequency cepstrum by using a preset speaker encoder to obtain sample speaker features; the sample speaker features are used to represent the speaking features of the first speaker; performing posterior encoding on the sample speech linear spectrum by using a preset posterior encoder to obtain sample speech posterior features; the sample speech features are used to represent the speaking features of the sample speaker and the speech content features of the sample speech content; performing content feature extraction according to the sample speaker features and the sample speech posterior features to obtain sample speech content features; performing speech synthesis according to preset second speaker features and sample speech content features to obtain sample synthesized speech data; the second speaker features are used to represent the speaking features of a second speaker, and the synthesized speech data is used to represent speech uttered by the second speaker according to the sample speech content. Performing spectral feature extraction on the sample synthesized speech data to obtain sample synthesized speech linear spectrum; performing loss calculation according to the sample synthesized speech linear spectrum and the sample original speech linear spectrum to obtain sample loss data; and performing parameter adjustment on the first encoder or the second encoder according to the sample second loss data to obtain the trained speaker encoder or the posterior encoder.

[0090] In step S103 of some embodiments, the original speech mel-frequency cepstrum is feature encoded by a preset speaker encoder to obtain original speaker features; wherein the original speaker features are used to represent the speaking features of the original speaker object. Specifically, the original speech mel-frequency cepstrum is taken as the network input of the speaker encoder, and the corresponding original speaker features are output.

[0091] In some embodiments, referring to Figure 3 The speaker encoder includes a first convolutional layer, a nonlinear neural network, and step S103 specifically includes but is not limited to steps S201 to S202:

[0092] In step S201, the original speech mel-frequency cepstrum is hidden state feature extracted by the first convolutional layer to obtain first hidden state speaker features;

[0093] In step S202, the first hidden state speaker features are processed in time sequence length by the nonlinear neural network to obtain the original speaker features.

[0094] Specifically, the first convolutional layer is used to extract the mel-frequency cepstrum hidden state to obtain the first hidden state speaker features. In an example, the first convolutional layer includes at least one convolution kernel, the convolution kernel size is 5, the channel number is 128, and the dimension is T*128. In order to obtain the speaker features irrelevant to the time sequence length and at the sentence level, the hidden state features are calculated along the time sequence dimension by using the nonlinear neural network, for example, the output of the last time step T is taken as the original speaker features, and the feature dimension can be 1*256. It should be noted that the nonlinear neural network of the embodiments of the present application can be a long short-term memory neural network (LSTM), or other nonlinear networks based on the LSTM neural network. For example, the nonlinear neural network can also be a bidirectional long short-term memory network (BILSTM), and the embodiments of the present application are not limited specifically.

[0095] In some embodiments, referring to Figure 4 The nonlinear neural network includes a residual layer, a second convolutional layer, and a nonlinear subnetwork, and step S202 specifically includes but is not limited to steps S301 to S303:

[0096] In step S301, the first hidden state speaker features are hidden state feature activated by the residual layer to obtain target activated speaker features;

[0097] In step S302, the target activated speaker features are hidden state feature extracted by the second convolutional layer to obtain second hidden state speaker features;

[0098] In step S303, the second hidden state speaker feature is processed in time sequence length by a nonlinear subnetwork to obtain the original speaker feature.

[0099] The steps S301 to S303 shown in the embodiments of the present application further extract the speaker feature of the first hidden state speaker feature by the residual layer and the second convolutional layer to obtain the second hidden state speaker feature which is more consistent with the original speaker feature of the speaker object. The nonlinear subnetwork is used to process the time sequence length to obtain the original speaker feature which is irrelevant to the time sequence length. It should be noted that the nonlinear subnetwork can be an LSTM neural network or a BILSTM neural network, which is not limited in the embodiments of the present application.

[0100] In an example, the second convolutional layer includes at least one convolutional kernel for extracting the mel cepstral hidden state, the size of the convolutional kernel is 7, the channel number is 256, and the dimension is T*256.

[0101] In some embodiments, referring to Figure 5 The residual layer includes the first convolutional kernel, the second convolutional kernel and the target activation function, and the step S301 specifically includes but is not limited to steps S401 to S404:

[0102] In step S401, the first hidden state speaker feature is extracted by the first convolutional kernel to obtain the first intermediate hidden state feature;

[0103] In step S402, the first hidden state speaker feature is processed by the target activation function for nonlinear transformation to obtain the target nonlinear hidden state feature;

[0104] In step S403, the target nonlinear hidden state feature is extracted by the second convolutional kernel to obtain the second intermediate hidden state feature;

[0105] In step S404, the first intermediate hidden state feature and the second intermediate hidden state feature are fused to obtain the target activation speaker feature.

[0106] The steps S401 to S404 shown in the embodiments of the present application perform deep extraction of the speaker feature of the first hidden state speaker feature by the first convolutional kernel, the second convolutional kernel and the target activation function, and the target activation speaker feature obtained has more potential speaker features of the original speaker object, thereby improving the similarity between the original speaker feature and the speaker feature of the original speaker object, and making the original speaker feature more capable of representing the speaker feature of the original speaker object.

[0107] In some embodiments, the target activation function includes a first activation function and a second activation function, and the step S402 includes:

[0108] The first hidden state feature is nonlinearly transformed by a first activation function to obtain a first nonlinear hidden state feature.

[0109] The first hidden state feature is nonlinearly transformed by a second activation function to obtain a second nonlinear hidden state feature.

[0110] The first nonlinear hidden state feature and the second nonlinear hidden state feature are point multiplied to obtain a target nonlinear hidden state feature.

[0111] Specifically, the nonlinear transformation of the feature by the first activation function and the second activation function and the point multiplication make the target nonlinear hidden state feature more conducive to obtaining the speaking feature of the original speaker. The first activation function is a hyperbolic tangent activation function (tanh), and the second activation function is a linear activation function (sigmoid).

[0112] It should be noted that since the original speech mel-spectrogram generally includes low-dimensional features, the low-dimensional features can be used to distinguish the speaker. Therefore, the original speaker feature of the original speech data has been obtained by extracting the speaking feature from the original speech mel-spectrogram. However, the speech content feature of the original speech data cannot be obtained. The content feature of the original speech data can be extracted by using an existing feature extraction model, but since the model is generally applicable, the performance of the model cannot be controlled when it is used for different original speech data, which may affect the accuracy of the content feature extraction and thus affect the effect of the speaker conversion. Therefore, the embodiment of the present application proposes to first encode the original speech linear spectrum by using a posterior feature encoder to obtain a speech posterior feature. The speech posterior feature can be used to represent the speaking feature of the original speaker and the speech content feature of the original speech content, and is also convenient for subsequent extraction of the content feature.

[0113] In step S104 of some embodiments, the original speech linear spectrum is encoded by using a preset posterior encoder to obtain a speech posterior feature; the speech posterior feature is used to represent the speaking feature of the original speaker and the speech content feature of the original speech content.

[0114] Specifically, the posterior encoder is used to encode the original speech linear spectrum to obtain a speech posterior feature. Since the original speech linear spectrum includes the low-frequency part and the high-frequency part of the original speech data, the original speech linear spectrum includes the speaking information of the original speaker and the content information of the original speech content, so that the resulting speech posterior feature can be used to represent the speaking feature of the original speaker and the speech content feature of the original speech content.

[0115] In step S105 of some embodiments, content feature extraction is performed according to the original speaker feature and the speech posterior feature, to obtain a target speech content feature.

[0116] In some embodiments, referring to Figure 6 , step S105 specifically includes but is not limited to steps S501-S502:

[0117] In step S501, a preset content feature extraction model is used to perform feature extraction on the speech posterior feature, to obtain a mapping content feature.

[0118] In step S502, the mapping content feature is adjusted according to the original speaker feature, to obtain a target content feature.

[0119] The steps S501-S502 shown in the embodiments of the present application perform feature extraction on the speech posterior feature by using a content feature extraction model. Since the speech posterior feature is obtained by performing feature encoding on the speech posterior feature by using a posterior feature encoder, the mapping content feature obtained can basically represent the original speech content of the original speech data, but may also include part of the speaker information. Therefore, the embodiments of the present application further adjust the mapping content feature by using the original speaker feature, to obtain a target content feature irrelevant to the speaker feature, and the target content feature can better represent the original speech content.

[0120] It should be noted that the content feature extraction model can be an existing content feature encoder, which is used to extract the content feature in the speech posterior feature, to obtain the mapping content feature.

[0121] In some embodiments, referring to Figure 7 , before step S501, the speech synthesis method of the embodiments of the present application further includes:

[0122] Training the content feature extraction model, specifically including but not limited to steps S601-S605:

[0123] In step S601, a preset original feature extraction model is used to perform feature extraction on the speech posterior feature, to obtain an initial speech content feature.

[0124] In step S602, speech synthesis is performed according to the original speaker feature and the initial speech content feature, to obtain intermediate speech data.

[0125] In step S603, spectral feature extraction is performed on the intermediate speech data, to obtain an intermediate speech linear spectrum.

[0126] In step S604, loss calculation is performed according to the intermediate speech linear spectrum and the original speech linear spectrum, to obtain loss data.

[0127] Step S605: adjusting parameters of the original feature extraction model according to the loss data to obtain a content feature extraction model.

[0128] The steps S601 to S605 shown in the embodiments of the present application are as follows: the original speech data is used to represent the speech uttered by the original speaker according to the original speech content, and the intermediate speech data is used to represent the speech uttered by the original speaker according to the initial speech content. The higher the similarity between the intermediate speech linear spectrum and the original speech linear spectrum, the more similar the initial speech content is to the original speech content. Therefore, in the embodiments of the present application, the loss data is used as a variable to adjust the parameters of the original feature extraction model, so that the loss data tends to converge, and the content feature extraction model is obtained. The content feature extraction model is used to extract features from the speech posterior features, and the mapping content features with high similarity to the original content features are obtained.

[0129] It should be noted that the original feature extraction model can be a shallow neural network or a deep neural network. It can include one content feature encoding layer or multiple content feature encoding layers. Each content feature encoding layer has corresponding network weights, and the content feature encoding layer is adjusted by adjusting the network weights to obtain the content feature extraction model of the embodiments of the present application.

[0130] In an embodiment, step S502 specifically includes: performing feature conversion on the original speaker feature by using a preset feature conversion function to obtain converted content features; and removing features related to the converted content features from the mapping content features to obtain target content features.

[0131] In another embodiment, the mapping content features include multiple content sub-features, and step S502 specifically includes: performing attention score calculation on each content sub-feature by using an attention model to obtain initial attention scores; if the similarity between the original speaker feature and the content sub-feature is greater than a preset threshold, the attention score is reduced to obtain a target attention score; if the similarity between the original speaker feature and the content sub-feature is less than or equal to the preset threshold, the attention score is increased to obtain a target attention score; and performing feature fusion on the multiple content sub-features according to the target attention scores to obtain target content features.

[0132] In step S106 of some embodiments, speech synthesis is performed according to the preset target speaker feature and target speech content feature to obtain synthesized speech data; wherein the target speaker feature is used to represent the speaking feature of the target speaker, and the synthesized speech data is used to represent the speech uttered by the target speaker according to the original speech content.

[0133] Specifically, the target speaker feature and the target speech content feature are spliced to obtain a spliced feature, and the spliced feature is input into a preset decoder to perform feature decoding processing on the spliced feature by the decoder to obtain the synthesized speech data.

[0134] Please refer to Figure 8 The embodiment of the present application also provides a speech synthesis device, which can implement the speech synthesis method. Figure 8 A module structure block diagram of the speech synthesis device provided by the embodiment of the present application is shown in the figure, which comprises a speech data acquisition module 701, a spectrum extraction module 702, a speaker feature encoding module 703, a posterior feature encoding module 704, a content feature extraction module 705 and a speech synthesis module 706. The speech data acquisition module 701 is configured to acquire original speech data; the original speech data is configured to represent a speech uttered by an original speaker according to a preset original speech content; the spectrum extraction module 702 is configured to perform spectrum feature extraction on the original speech data to obtain original speech mel-cepstrum and original speech linear spectrum; the speaker feature encoding module 703 is configured to perform feature encoding on the original speech mel-cepstrum by a preset speaker encoder to obtain original speaker features; the original speaker features are configured to represent the speaking features of the original speaker; the posterior feature encoding module 704 is configured to perform posterior encoding on the original speech linear spectrum by a preset posterior encoder to obtain speech posterior features; the speech posterior features are configured to represent the speaking features of the original speaker and the speech content features of the original speech content; the content feature extraction module 705 is configured to perform content feature extraction according to the original speaker features and the speech posterior features to obtain target speech content features; and the speech synthesis module 706 is configured to perform speech synthesis according to preset target speaker features and target speech content features to obtain synthesized speech data; the target speaker features are configured to represent the speaking features of a target speaker, and the synthesized speech data is configured to represent a speech uttered by the target speaker according to the original speech content.

[0135] It should be noted that the specific implementation of the speech synthesis device is basically the same as the specific embodiment of the speech synthesis method described above, and will not be repeated here.

[0136] The embodiment of the present application also provides an electronic device, which comprises a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory, and the program is executed by the processor to implement the speech synthesis method. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer and the like.

[0137] Please refer to Figure 9 , Figure 9 The hardware structure of the electronic device of another embodiment is shown, which comprises:

[0138] The processor 801 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0139] The memory 802 can be implemented by a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory 802 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 802 and are called and executed by the processor 801 to implement the voice synthesis method of the embodiments of the present application.

[0140] The input / output interface 803 is configured to implement information input and output.

[0141] The communication interface 804 is configured to implement the communication interaction between the device and other devices, and can realize the communication through a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0142] The bus 805 is configured to transmit information between various components (for example, the processor 801, the memory 802, the input / output interface 803, and the communication interface 804) of the device.

[0143] The processor 801, the memory 802, the input / output interface 803, and the communication interface 804 are connected to each other through the bus 805 to realize the communication connection between the device.

[0144] The embodiments of the present application further provide a storage medium, which is a computer readable storage medium, is configured to store computer readable information, and stores one or more programs. The one or more programs can be executed by one or more processors to implement the voice synthesis method.

[0145] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory that is remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The speech synthesis method, the speech synthesis device, the electronic equipment and the storage medium provided by the embodiments of the present application can obtain the original speech mel-frequency cepstrum and the original speech linear spectrum by performing spectrum feature extraction on the original speech data. The original speech mel-frequency cepstrum is used to represent the speaking feature information of the original speaker object. The original speech linear spectrum is used to represent the speaking feature information of the original speaker object and the speech feature information of the original speech content. Therefore, the original speaker feature can be obtained by performing feature coding on the original speech mel-frequency cepstrum by the speaker encoder. The speech posterior feature can be obtained by performing feature coding on the original speech linear spectrum by the posterior feature encoder. The target speech content feature irrelevant to the speaker feature can be obtained by performing content feature extraction on the original speaker feature and the speech posterior feature, which is equivalent to removing the features related to the original speaker feature from the speech posterior feature. Finally, the synthesized speech data can be obtained by performing speech synthesis according to the target speaker feature and the target speech content feature. In summary, the embodiments of the present application can realize speech synthesis and speech conversion for any speaker. In addition, the embodiments of the present application do not need to use the speech data of the target speaker object to participate in model fine-tuning, and also improve the efficiency of speech conversion.

[0147] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0148] Those skilled in the art can understand that, Figures 2-7 The technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0149] The apparatus embodiments described above are merely exemplary, and the units described as separate units can or can not be physically separate, i.e., can be located in one place, or can be distributed over multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0150] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof.

[0151] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims of the foregoing drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so

[0152] It should be understood that in this application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases: only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are in an "or" relationship. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c, can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0153] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0154] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed to a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0155] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0156] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the technical solutions or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0157] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modification, equivalent replacement and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A speech synthesis method, characterized in that, The method includes: Acquire raw speech data; the raw speech data is used to represent the speech produced by the original speaker according to the preset raw speech content; Spectral features are extracted from the original speech data to obtain the original speech Mel-cepstrum and the original speech linear spectrum; The original speech is feature-encoded using a preset speaker encoder to obtain original speaker features; wherein, the original speaker features are used to represent the speaking characteristics of the original speaking object, and the speaker encoder includes: a first convolutional layer, a residual layer, a second convolutional layer, and a nonlinear subnetwork; The original speech linear spectrum is posteriorly encoded by a preset posterior encoder to obtain speech posterior features; the speech posterior features are used to represent the speech features of the original speaker and the speech content features of the original speech content. Content features are extracted based on the original speaker features and the posterior speech features to obtain the target speech content features; Speech synthesis is performed based on preset target speaker features and target speech content features to obtain synthesized speech data; wherein, the target speaker features are used to represent the speaking characteristics of the target speaking object, and the synthesized speech data is used to represent the speech produced by the target speaking object based on the original speech content; The step of performing feature encoding on the original speech Mel-cepstrum using a preset speaker encoder to obtain the original speaker features includes: The first hidden state speaker features are obtained by extracting the Mel cepstral spectra of the original speech through the first convolutional layer. The target activated speaker features are obtained by activating the hidden state features of the first hidden state speaker features through the residual layer. The second convolutional layer is used to extract the latent state features of the target activated speaker to obtain the second latent state speaker features. The original speaker features are obtained by performing temporal length processing on the second hidden state speaker features through the nonlinear subnetwork.

2. The method according to claim 1, characterized in that, The step of extracting content features based on the original speaker features and the posterior speech features to obtain target speech content features includes: The posterior features of the speech are extracted using a preset content feature extraction model to obtain the mapped content features; The target speech content features are obtained by adjusting the mapped content features based on the original speaker features.

3. The method according to claim 2, characterized in that, Before extracting features from the posterior speech features using a preset content feature extraction model to obtain the mapped content features, the method further includes: Training the content feature extraction model specifically includes: The speech posterior features are extracted using a preset original feature extraction model to obtain initial speech content features; Speech synthesis is performed based on the original speaker features and the initial speech content features to obtain intermediate speech data; Spectral features are extracted from the intermediate speech data to obtain the linear spectrum of the intermediate speech. Loss is calculated based on the intermediate speech linear spectrum and the original speech linear spectrum to obtain the first loss data; The original feature extraction model is adjusted based on the first loss data to obtain the content feature extraction model.

4. The method according to any one of claims 1 to 3, characterized in that, The residual layer includes a first convolutional kernel, a second convolutional kernel, and a target activation function. The step of activating the hidden state features of the first hidden state speaker features through the residual layer to obtain the target activated speaker features includes: The first hidden state features are obtained by extracting features from the first hidden state speaker features using the first convolutional kernel; The first hidden state speaker features are processed by nonlinear transformation using the target activation function to obtain the target nonlinear hidden state features. The second intermediate hidden state features are obtained by extracting features from the target nonlinear hidden state features using the second convolution kernel. The first intermediate hidden state feature and the second intermediate hidden state feature are fused to obtain the target active speaker feature.

5. The method according to claim 4, characterized in that, The target activation function includes a first activation function and a second activation function. The step of performing a nonlinear transformation on the first hidden state features using the target activation function to obtain target nonlinear hidden state features includes: The first nonlinear hidden state features are obtained by performing a nonlinear transformation on the first hidden state features using the first activation function. The second nonlinear hidden state features are obtained by performing a nonlinear transformation on the first hidden state features using the second activation function. The first nonlinear hidden state feature and the second nonlinear hidden state feature are multiplied by a dot product to obtain the target nonlinear hidden state feature.

6. A speech synthesis device, characterized in that, The device includes: The voice data acquisition module is used to acquire raw voice data; the raw voice data represents the voice produced by the original speaker according to the preset raw voice content. The spectrum extraction module is used to extract spectral features from the original speech data to obtain the original speech Mel-cepstrum and the original speech linear spectrum; The speaker feature encoding module is used to perform feature encoding on the original speech Mel-Cepstral spectrum through a preset speaker encoder to obtain the original speaker features; wherein, the original speaker features are used to represent the speaking features of the original speaking object, and the speaker encoder includes: a first convolutional layer, a residual layer, a second convolutional layer, and a nonlinear subnetwork; The posterior feature encoding module is used to perform posterior encoding on the original speech linear spectrum through a preset posterior encoder to obtain speech posterior features; the speech posterior features are used to represent the speech features of the original speaker and the speech content features of the original speech content. The content feature extraction module is used to extract content features based on the original speaker features and the posterior speech features to obtain the target speech content features; The speech synthesis module is used to synthesize speech based on preset target speaker features and target speech content features to obtain synthesized speech data; wherein, the target speaker features are used to represent the speaking characteristics of the target speaking object, and the synthesized speech data is used to represent the speech produced by the target speaking object based on the original speech content; The step of performing feature encoding on the original speech Mel-cepstrum using a preset speaker encoder to obtain the original speaker features includes: The first hidden state speaker features are obtained by extracting the Mel cepstral spectra of the original speech through the first convolutional layer. The target activated speaker features are obtained by activating the hidden state features of the first hidden state speaker features through the residual layer. The second convolutional layer is used to extract the latent state features of the target activated speaker to obtain the second latent state speaker features. The original speaker features are obtained by performing temporal length processing on the second hidden state speaker features through the nonlinear subnetwork.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the speech synthesis method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech synthesis method according to any one of claims 1 to 5.

Images