Prosody encoder training method, speech conversion method and related products

Abstract

An embodiment of the present invention provides a method for training a prosody encoder, comprising: obtaining multi-level coding features of speech training data from the training data based on a speech recognition model; extracting the multi-level coding features based on multiple coding output layers of the encoder of the speech recognition model; and training the prosody encoder using the multi-level coding features of the training data, so that the trained prosody encoder can be used to extract prosody information. The method of the present invention enables the prosody encoder to accurately extract prosody features, thereby significantly improving the speech conversion effect and providing a better user experience. Furthermore, embodiments of the present invention provide a speech conversion method, a prosody encoder training device, a speech conversion device, an apparatus, and a computer-readable storage medium.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of information processing technology. More specifically, the embodiments of the present invention relate to a prosody encoder training method, a speech conversion method, a prosody encoder training device, a speech conversion device, an apparatus for executing the aforementioned method, and a computer-readable storage medium. Background Art

[0002] This section is intended to provide background or context for embodiments of the present invention as recited in the claims. The description herein may include concepts that could be explored, but not necessarily concepts that have been previously conceived or explored. Therefore, unless otherwise indicated herein, the material described in this section is not prior art with respect to the specification and claims of this application and is not admitted to be prior art by inclusion in this section.

[0003] A key aspect of speech conversion is transferring the linguistic content from a source speaker to a target speaker. Deep learning can enable many speech conversion scenarios: The first is speech conversion with non-parallel data from paired speakers. Speech conversion with non-parallel training data is a task similar to image-to-image conversion. The second is decoupling the speaker from the linguistic content, distinguishing between speaker identity and linguistic content. In the context of speech conversion, speech can be considered a combination of speaker identity and linguistic content.

[0004] The third is to use a text-to-speech (TTS) system for speech conversion. The fourth is to use Automatic Speech Recognition (ASR) technology for speech conversion. Speech synthesis (TTS), also known as text-to-speech technology, is a technology that converts text information into understandable and fluent speech output. The TTS system provides a mechanism for speech to follow the language content. The idea of ​​using the TTS mechanism can be motivated in different ways. It is the opposite of the goal of speech recognition technology we are more familiar with. ASR converts sound into text, which is analogous to the human ear. TTS, on the other hand, converts text into speech (reading it aloud), which is analogous to the human mouth.

[0005] With the development of artificial intelligence (AI) technology, text-to-speech (TTS) is widely used in educational products. However, during the speech synthesis process, it is difficult to fine-tune the duration, emphasis, rhythm, and emotion of the pronunciation. These difficult-to-model phonemes result in poor speech synthesis performance. However, speech conversion can control the original speaker's speaking rhythm and precisely customize the target voice, offering advantages in some scenarios. The difference between speech conversion and speech synthesis tasks is that speech conversion converts the original speech into the voice of a specified speaker, while speech synthesis converts text into speech.

[0006] Currently, when using deep learning models for speech conversion, the speech's mel spectrum undergoes multiple layers of encoding in speech recognition. Due to bottleneck features, this process can lose information such as the original speaker's intonation, prosody, and emotion. To enhance the modeling of the converted speech's prosodic information, a common approach is to use a deep neural network to extract prosodic features from the mel spectrum / wav2vec output features. However, extracting prosodic features from the wav2vec output requires a large, unsupervised speech model to extract compressed speech features, resulting in some prosodic loss. Furthermore, feature extraction from this model is challenging and difficult to implement. Furthermore, the prosodic features output by this approach contain rich speaker information, which can easily affect the speech conversion performance if not processed.

[0007] Based on this, how to solve the problem that prosodic feature extraction is difficult and susceptible to interference in the existing speech conversion process plays an important role in improving the effect of speech conversion. Summary of the Invention

[0008] The known process of voice conversion does not achieve satisfactory prosodic feature extraction, which is a very annoying process.

[0009] Therefore, there is a great need for an improved prosody encoder training method, speech conversion method and related products, so that the prosody encoder can extract prosody information with high fidelity, so that the converted speech can better retain the prosody-related information of the original speaker.

[0010] In this context, embodiments of the present invention are intended to provide a prosody encoder training method and related products.

[0011] In a first aspect of the embodiments of the present invention, a method for training a prosody encoder is provided, comprising: obtaining multi-level coding features of speech training data from the training data based on a speech recognition model; extracting the multi-level coding features based on multiple coding output layers of an encoder of the speech recognition model; and training the prosody encoder using the multi-level coding features of the training data, so as to use the trained prosody encoder for extracting prosody information.

[0012] In one embodiment of the present invention, the speech recognition model includes an encoder, the encoder has multiple encoding output layers, each encoding output layer is used to output single-level encoding features, and the multi-level encoding features of the training data obtained from the training data based on the speech recognition model include: inputting the Mel spectrum of the training data into the speech recognition model; extracting multiple single-level encoding features from the multiple encoding output layers of the encoder in the speech recognition model; and generating multi-level encoding features based on the multiple single-level encoding features.

[0013] In another embodiment of the present invention, the prosody encoder integrates a channel attention mechanism and a spatial attention mechanism to perform channel attention weighting and spatial attention weighting on the multi-level encoding features.

[0014] In another embodiment of the present invention, the prosody encoder includes a channel attention module and a spatial attention module, wherein channel attention weighting and spatial attention weighting of the multi-level coding features include: using the channel attention module to calculate the channel feature weight distribution corresponding to the multi-level coding features of the training data to perform channel attention weighting; using the spatial attention module to calculate the spatial feature weight distribution corresponding to the multi-level coding features after channel attention weighting to perform spatial attention weighting.

[0015] In another embodiment of the present invention, the use of the channel attention module to calculate the channel feature weight distribution corresponding to the multi-level coding features of the training data to perform channel attention weighting includes: multiplying the channel feature weights with the multi-level coding features respectively to obtain the multi-level coding features for updating the channel weights.

[0016] In one embodiment of the present invention, the spatial feature weight distribution corresponding to the multi-level coding features after channel attention weighting is calculated using the spatial attention module to perform spatial attention weighting includes: multiplying the spatial feature weights with the multi-level coding features of the updated channel weights to obtain the coding information output by the prosody encoder.

[0017] In another embodiment of the present invention, the training of the prosody encoder using the multi-level coding features of the training data includes: the prosody encoder obtaining the coding information output by the prosody encoder based on the multi-level coding features of the training data; and performing gradient reversal adversarial training on the prosody encoder to remove residual speaker information in the coding information output by the prosody encoder, thereby extracting the prosody information of the training data.

[0018] In another embodiment of the present invention, the speech training data includes speaker labels, wherein the adversarial training includes: training a speaker classifier using the encoding information output by the prosody encoder and the speaker labels contained in the training data to obtain a classification result; and reversely updating the weights of the neural network in the prosody encoder based on the classification result to complete gradient reversal, thereby removing residual speaker information in the encoding information output by the prosody encoder.

[0019] In a second aspect of the embodiments of the present invention, the present invention provides a training device for a prosody encoder, comprising: a speech recognition module configured to obtain multi-level coding features of the training data from speech training data; the multi-level coding features are extracted based on multiple coding output layers of an encoder of a speech recognition model; a prosody encoder connected to the speech recognition module and configured to perform training using the multi-level coding features of the training data so as to use the trained prosody encoder for extracting prosody information.

[0020] In a third aspect of the embodiments of the present invention, the present invention also provides a method for speech conversion, comprising: obtaining bottleneck features and multi-level coding features of speech from the speech to be converted based on a speech recognition model, wherein the multi-level coding features are extracted based on multiple coding output layers of the encoder of the speech recognition model; extracting speech content information from the bottleneck features using a content encoder; extracting prosody information from the multi-level coding features using a trained prosody encoder, wherein the trained prosody encoder is trained based on the methods described above and in multiple embodiments below; and inputting the prosody information, speech content information, and target speaker label into a decoder to complete the speech conversion.

[0021] In one embodiment of the present invention, the speech recognition model includes an encoder, the encoder has multiple encoding output layers, each encoding output layer is used to output single-level encoding features, and the bottleneck features and multi-level encoding features of the speech to be converted are obtained from the speech to be converted based on the speech recognition model, including: inputting the Mel spectrum of the speech to be converted into the speech recognition model; extracting multiple single-level encoding features from the multiple encoding output layers of the encoder in the speech recognition model; generating multi-level encoding features based on the multiple single-level encoding features, and using the output of the last encoding output layer as the bottleneck feature.

[0022] In another embodiment of the present invention, inputting prosody information, speech content information, and a target speaker label into a decoder to complete speech conversion includes: fusing the prosody information, speech content information, and target speaker label to decode and generate spectral features; and synthesizing speech based on the spectral features to achieve speech conversion.

[0023] In yet another embodiment of the present invention, synthesizing speech according to the spectral features to achieve speech conversion includes: utilizing a vocoder to restore a speech waveform from the spectral features to synthesize speech.

[0024] In a fourth aspect of the embodiments of the present invention, the present invention further provides a speech conversion device, comprising: a speech recognition module, configured to obtain bottleneck features and multi-level coding features of speech from the speech to be converted, wherein the multi-level coding features are extracted based on multiple coding output layers of an encoder of a speech recognition model; a content encoder, configured to extract speech content information from the bottleneck features; a prosody encoder, configured to extract prosody information from the multi-level coding features, wherein the prosody encoder is trained based on the methods described in the foregoing and subsequent embodiments; and a decoder, connected to the speech recognition module, the content encoder, and the prosody encoder, and configured to complete speech conversion based on the prosody information, speech content information, and a target speaker label.

[0025] In a fifth aspect of the embodiments of the present invention, a device is provided, comprising: a processor; and a memory storing computer instructions for training a prosody encoder, wherein when the computer instructions are executed by the processor, the device executes the prosody encoder training method described in the foregoing and following embodiments; the memory further stores computer instructions for speech conversion, wherein when the computer instructions are executed by the processor, the device executes the speech conversion method described in the foregoing and following embodiments.

[0026] In a sixth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, comprising program instructions for training a prosody encoder or performing speech conversion. When the program instructions are executed by a processor, the prosody encoder training method described in the foregoing and following embodiments is implemented, or when the program instructions are executed by a processor, the speech conversion method described in the foregoing and following embodiments is implemented.

[0027] According to an embodiment of the present invention, a prosody encoder training method and related products can train the prosody encoder based on the multi-level coding features of training data. This allows the prosody encoder to extract richer prosodic information when used for speech conversion, thereby reducing prosodic loss during the speech conversion process, effectively improving the speech conversion effect, and providing a better user experience. In some embodiments, when training the prosody encoder, a channel attention mechanism and a spatial attention mechanism are integrated into the prosody encoder to selectively weight the extracted prosodic features of speech at different levels, so that the converted speech can better retain the prosodic information of the original speaker.

[0028] Furthermore, the embodiment of the present disclosure also performs gradient reversal adversarial training on the prosody encoder during the training process, thereby effectively removing the residual speaker information in the encoded information output by the prosody encoder, reducing interference with the speech conversion results, and thus achieving high-fidelity prosody representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily apparent by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present invention are shown by way of example and not limitation, in which:

[0030] Figure 1 A block diagram schematically illustrates an exemplary computing system 100 suitable for implementing embodiments of the present invention;

[0031] Figure 2 Schematically shows a flow chart of a method for training a prosody encoder according to one embodiment of the present invention;

[0032] Figure 3 Schematically shows a flow chart of a method for training a prosody encoder according to another embodiment of the present invention;

[0033] Figure 4 The following schematically illustrates a multi-level coding feature extraction process according to an embodiment of the present invention;

[0034] Figure 5 Schematically shows a schematic diagram of performing channel attention and spatial attention weighting on multi-level coding features according to one embodiment of the present invention;

[0035] Figure 6 Schematically shows a schematic diagram of an attention weighting process according to one embodiment of the present invention;

[0036] Figure 7 The following schematically shows a flow chart of a method for voice conversion according to an embodiment of the present invention;

[0037] Figure 8 Schematically shows a schematic block diagram of a prosody encoder training device according to an embodiment of the present invention;

[0038] Figure 9 A schematic block diagram of a speech conversion device according to an embodiment of the present invention is schematically shown;

[0039] Figure 10 A schematic block diagram of a device according to an embodiment of the present invention is schematically shown.

[0040] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts. DETAILED DESCRIPTION

[0041] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0042] Figure 1 1 shows a block diagram of an exemplary computing system 100 suitable for implementing embodiments of the present invention. Figure 1 As shown, the computing system 100 may include a central processing unit (CPU) 101, a random access memory (RAM) 102, a read-only memory (ROM) 103, a system bus 104, a hard disk controller 105, a keyboard controller 106, a serial interface controller 107, a parallel interface controller 108, a display controller 109, a hard disk 110, a keyboard 111, a serial peripheral device 112, a parallel peripheral device 113, and a display 114. Of these devices, the CPU 101, RAM 102, ROM 103, hard disk controller 105, keyboard controller 106, serial controller 107, parallel controller 108, and display controller 109 are coupled to the system bus 104. The hard disk 110 is coupled to the hard disk controller 105, the keyboard 111 is coupled to the keyboard controller 106, the serial peripheral device 112 is coupled to the serial interface controller 107, the parallel peripheral device 113 is coupled to the parallel interface controller 108, and the display 114 is coupled to the display controller 109. It should be understood that Figure 1 The structured block diagram is only for the purpose of illustration, rather than for limiting the scope of the present invention. In some cases, some devices may be added or reduced according to specific circumstances.

[0043] Those skilled in the art will appreciate that embodiments of the present invention may be implemented as a system, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the present invention may also be implemented in the form of a computer program product in one or more computer-readable media containing computer-readable program code.

[0044] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive examples) of computer-readable storage media can include, for example: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0045] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0046] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0047] The computer program code for performing the operations of the present invention can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0048] The following will describe the embodiments of the present invention with reference to the flowcharts of the methods and block diagrams of the devices (or systems) according to the embodiments of the present invention. It should be understood that each block in the flowcharts and / or block diagrams, as well as the combination of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, and these computer program instructions are executed by the computer or other programmable data processing device to produce a device that implements the functions / operations specified in the blocks in the flowcharts and / or block diagrams.

[0049] These computer program instructions can also be stored in a computer-readable medium that enables a computer or other programmable data processing device to operate in a specific manner. In this way, the instructions stored in the computer-readable medium produce a product that includes an instruction device that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0050] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer or other programmable apparatus provide a process that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0051] According to an embodiment of the present invention, a prosody encoder training method and related products are proposed.

[0052] In this article, it is important to understand that the terms involved are:

[0053] Voice Conversion (VC): refers to the process of converting a person's voice to sound like another person while preserving the content of the speech.

[0054] Mel spectrum: a common compression method for speech. Mel spectrum is a spectrum converted from frequency to mel scale. According to the characteristics of human hearing, we are more sensitive to low-frequency sounds and less sensitive to high-frequency sounds. So when the sound frequency increases linearly, the higher the frequency, the harder it is for us to hear the difference. Therefore, a logarithmic spectrum is used instead of a linear spectrum. The core of the Mel spectrum is the Mel-scale, a logarithmic scale for linear changes in frequency perception. Humans are more sensitive to frequency on the mel scale. The Mel spectrum mainly includes three major characteristics: time-frequency domain information, perception-related amplitude information, and perception-related frequency domain information. Various applications of the Mel spectrum mainly include the following types: audio classification, automatic emotion recognition, music genre classification, and music instrument classification. The embodiments of this disclosure mainly describe the application of the Mel spectrum in a prosody encoder.

[0055] CBAM (Convolutional block attention module) attention mechanism: The CBAM attention mechanism is divided into two parts: spatial attention and channel attention.

[0056] Channel attention: By modeling the importance of each feature channel, different channels can be enhanced or suppressed for different tasks.

[0057] Spatial attention: Not all areas contribute equally to the task. Only task-related areas need to be cared about, such as the main body of the classification task. The spatial attention model is to find the most important parts of the network for processing.

[0058] In addition, any number of elements in the drawings is for illustration and not limitation, and any naming is for distinction only and does not have any limiting meaning.

[0059] The principles and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention. SUMMARY OF THE INVENTION

[0061] The inventors have discovered that existing prosody encoders are unable to effectively extract the speaker's prosodic information, resulting in poor speech conversion results. Specifically, in speech conversion systems, bottleneck features are generated after the mel spectrum undergoes multi-layer encoding by speech recognition. Because bottleneck features lose information such as the original speaker's intonation, prosody, and emotion after multi-layer encoding by speech recognition, and because the output features of the mel spectrum or wav2vec contain rich speaker information, the prosody encoder is unable to extract rich and effective prosodic information from the mel spectrum or wav2vec output features, resulting in unclear prosodic representation and an inability to effectively preserve the original speaker's prosodic information in the converted speech.

[0062] Based on this, the inventors discovered that the key to fully preserving the prosodic information in speech lies in enhancing the prosodic modeling of the original speech by the converted speech, enabling the prosodic encoder to output richer prosodic information. Specifically, this can be achieved by acquiring multi-layered coding features from the speech recognition model's encoder. The rich prosodic information in these multi-layered coding features can then be used to train the prosodic encoder, thereby improving its effectiveness in extracting prosodic information.

[0063] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention are described in detail below.

[0064] Exemplary Methods

[0065] Reference below Figure 2 The training method of the prosody encoder according to the exemplary embodiment of the present invention is described below. It should be noted that the embodiments of the present invention can be applied to any applicable scenario.

[0066] Figure 2 The flowchart of the prosody encoder training method 200 according to one embodiment of the present invention is schematically shown.

[0067] like Figure 2 As shown, at step S201, multi-level coding features of the training data are obtained from the training data of the speech based on the speech recognition model. The multi-level coding features can be extracted based on multiple coding output layers of the encoder of the speech recognition model. In some embodiments, the encoder of the speech recognition model includes multiple coding output layers, and there is a certain correlation between the coding output layers. Each coding output layer can output the coding features of the corresponding layer, and the proportion of the speaker's timbre, prosody and language content information contained in the coding features of each layer is different. Specifically, the higher the level of the feature, the more information about its timbre and prosody, and the less information about the language content. Based on this, by extracting multi-level coding features from the training data of the speech according to the speech recognition model, a richer and multi-scale prosodic representation can be obtained, and the multi-level coding features have excellent noise resistance.

[0068] At step S202, a prosody encoder is trained using the multi-level encoding features of the training data, so that the trained prosody encoder can be used to extract prosody information. In some embodiments, training the prosody encoder based on the multi-level encoding features of the training data and the prosody information corresponding to the training data can effectively improve the prosody encoder's ability to extract prosody information.

[0069] The following combination Figure 3 Some possible exemplary implementations of the above steps are further described.

[0070] Figure 3 Schematically shows a flow chart of a prosody encoder training method 300 according to another embodiment of the present invention. Figure 3 As shown, at step S301, the Mel spectrum of the training data is input into the speech recognition model. In some embodiments, the speech recognition model includes two parts: an encoder and a decoder. In this solution, only the encoder in the speech recognition model is used, and the input of the encoder is the Mel spectrum and the output is characters.

[0071] At step S302, multiple single-level coding features are extracted from multiple coding output layers of an encoder in the speech recognition model. In some embodiments, the encoder has multiple coding output layers, each of which is configured to output single-level coding features. Coding features at different levels have different characteristics. Higher-level features contain more information about timbre and prosody, but less information about language content.

[0072] At step S303, a multi-level coding feature is generated based on the multiple single-level coding features. In some embodiments, the corresponding multi-level coding features are obtained based on the multiple coding output layers of the encoder. The multi-level coding features can be used to obtain relatively accurate prosody information contained in the speech training data, which can be used to train the prosody encoder.

[0073] In some embodiments, the prosody encoder can integrate a channel attention mechanism and a spatial attention mechanism to perform channel attention weighting and spatial attention weighting on multi-level coding features. Based on this, the prosody encoder includes a channel attention module and a spatial attention module.

[0074] Specifically, at step S304, the channel attention module is used to calculate the channel feature weight distribution corresponding to the multi-level coding features of the training data to perform channel attention weighting. In some embodiments, each layer of coding features in the multi-level coding features has a certain connection, and each layer of coding features has different attention points. The channel attention module can obtain the channel attention weights of each layer of multi-level coding features to weight each layer of coding features separately, thereby improving the accuracy and efficiency of prosodic information.

[0075] In some application scenarios, when using the channel attention module to perform channel attention weighting, the channel feature weights can be multiplied with the multi-level coding features to obtain the multi-level coding features for updating the channel weights.

[0076] In step S305, the spatial feature weight distribution corresponding to the multi-level coding features after channel attention weighting is calculated using the spatial attention module to perform spatial attention weighting.

[0077] In some application scenarios, when using the spatial attention module to perform spatial attention weighting, the spatial feature weights can be multiplied with the multi-level coding features of the updated channel weights to obtain the coding information output by the prosody encoder.

[0078] At step S306, the prosody encoder obtains encoding information output by the prosody encoder based on the multi-level encoding features of the training data. In some embodiments, the prosody encoder is trained based on the multi-level encoding features of the training data. During the training process, encoding information output by the prosody encoder is obtained, and based on the output encoding information, it is determined whether the prosody encoder can extract valid prosody information.

[0079] In order to achieve high-quality prosodic representation, it is necessary to remove the speaker information contained in the prosodic features. Based on this, the present disclosure also performs gradient reversal training on the prosodic encoder to reduce the interference of the original speaker information in the prosodic features on the speech conversion results.

[0080] In step S307 , the prosody encoder is subjected to gradient reversal adversarial training to remove residual speaker information from the encoded information output by the prosody encoder, thereby extracting the prosody information of the training data.

[0081] In some embodiments, the speech training data includes speaker labels. During adversarial training with gradient reversal, the speaker classifier can be trained using the encoded information output by the prosody encoder and the speaker labels contained in the training data to obtain a classification result. Based on the classification result, the weights of the neural network in the prosody encoder are reversely updated to complete gradient reversal, thereby removing any residual speaker information from the encoded information output by the prosody encoder.

[0082] Next, we will describe in detail the extraction process of the above multi-level encoding features and the attention weighting process.

[0083] Figure 4 The following schematically illustrates a multi-level coding feature extraction process 400 according to an embodiment of the present invention. Figure 4 It can be combined with the previous text Figure 2 and Figure 3 An exemplary implementation of the steps described. Figure 2 and Figure 3 The detailed description of each step also applies below.

[0084] like Figure 4As shown, the end-to-end speech recognition system based on the comformer structure includes two modules, an encoder and a decoder. The input is Mel spectrum features and the output is characters. The disclosed embodiment only uses the encoder module of the speech recognition system. In the encoder, the stacked N-layer encoding output layer (such as the comformer module) will output N-dimensional features, that is, each layer of the comformer module will output a HEF feature (Hierarchical Encoder Feature, referred to as HEF), and multiple HEFs constitute a multi-level encoding feature. The feature of the highest layer of HEF is BNF (BottleNeck Feature, BNF). In the process of speech recognition, the Mel spectrum of the speech will first be encoded through a multi-layer Encoder (encoder) network, and then down-sampled decoding will be performed. The output of this layer where encoding ends and decoding begins is defined as BNF.

[0085] BNF and HEF are similar in that they both contain three pieces of information: speaker timbre, prosody, and linguistic content. However, HEF differs from BNF in that it includes richer prosodic information. Based on this, multiple encoder output layers in the encoder can output multi-level encoding features for use in training the prosodic encoder.

[0086] Figure 5 The figure schematically shows a schematic diagram of weighting channel attention and spatial attention for multi-level coding features according to an embodiment of the present invention. Figure 6 The following schematically shows a diagram of an attention weighting process according to an embodiment of the present invention. Figure 5 It can be combined with the previous text Figure 2 and Figure 3 An exemplary implementation of the steps described. Figure 6 It can also be combined with the previous text Figure 2 and Figure 3 An exemplary implementation of the steps described. Figure 2 and Figure 3 The detailed description of each step also applies below.

[0087] The prosody encoder is a unidirectional recurrent neural network, where the hidden state at each time step depends only on the input subsequence at that time step and before. A prosody encoder can also be constructed using a bidirectional recurrent neural network. In this case, the hidden state of the prosody encoder at each time step depends on both the subsequences before and after that time step (including the input at the current time step), encoding information about the entire sequence.

[0088] In an embodiment of the present disclosure, the prosody encoder includes a spectrum preprocessing network, an attention module, and a fully connected layer. The spectrum information is sequentially input into the spectrum preprocessing network, the attention module, and the fully connected layer to obtain the prosody features of the speech at the phoneme level. Specifically, the spectrum information can be the Mel spectrum in the above content, such as an 80-dimensional Mel spectrum. The attention module can include a channel attention module and a spatial attention module, which calculates the weighted sum of the high-dimensional spectrum features in the time dimension through the attention mechanism to obtain the attention features. The fully connected layer can calculate the weighted sum of the attention features in the feature dimension, and finally obtain the final prosody features of the speech at the phoneme level by averaging the output of the fully connected layer in the time period corresponding to the phoneme.

[0089] Furthermore, the prosody encoder includes a decoupling module that decouples the textual content information (or speaker information) and prosody information contained in the prosodic features of speech, leaving only the prosodic information of the prosodic features of speech. In this solution, the decoupling module is designed to generate an adversarial network to decouple the speaker information and prosody information contained in the prosodic features of speech. Since the adversarial training process has been described in detail above, it will not be repeated here.

[0090] like Figure 5 As shown in the figure, the attention mechanism (Convolutional block attention module, referred to as CBAM) adopted in this disclosure can be divided into two parts: spatial attention and channel attention. In the network layer, the channel attention weight is calculated first, and then the spatial attention weight is calculated. Based on this, this disclosure only calculates the channel attention weight first and then the spatial attention weight for illustrative purposes.

[0091] A channel attention module and a spatial attention module are set up in the prosody encoder. On the one hand, by modeling the importance of each feature channel, different channels are enhanced or suppressed for different tasks; on the other hand, important parts of the network model in the prosody encoder are found for processing to perform key area analysis on the multi-level coding features, thereby enhancing the parts that contribute to the output results and improving the effectiveness of the output results of the prosody encoder.

[0092] like Figure 6As shown, the present disclosure can perform channel attention weighting and spatial attention weighting on multi-level coding features respectively. Specifically, the channel attention module is used to calculate the channel feature weight distribution corresponding to the input multi-level coding features, and the multi-level coding features are multiplied respectively using the channel feature weights to realize the weighted calculation process. Then the spatial feature weight distribution of the multi-level coding features is calculated using the spatial attention module, and further spatial attention weighting of the multi-level encoder features is realized by multiplying it with the result obtained by the calculation of the aforementioned channel attention weighting process. Through two different attention weighting mechanisms, effective rhythmic features can be extracted from the multi-level coding features, thereby improving the performance of the rhythmic encoder.

[0093] In another aspect of the present invention, a voice conversion method is provided. Figure 7 The flowchart of the voice conversion method 700 according to one embodiment of the present invention is schematically shown.

[0094] like Figure 7 As shown, at step S701, bottleneck features and multi-level coding features of speech are obtained from the speech to be converted based on the speech recognition model. The multi-level coding features can be extracted based on multiple coding output layers of the encoder of the speech recognition model.

[0095] In some embodiments, the above-mentioned speech recognition model includes an encoder and a decoder. The encoder has multiple encoding output layers, and each encoding output layer is used to output single-level encoding features. When obtaining the bottleneck features and multi-level encoding features of the speech from the speech to be converted based on the speech recognition model, the following method can be used to obtain them: the Mel spectrum of the speech to be converted is input into the speech recognition model. Then, multiple single-level encoding features are extracted from the multiple encoding output layers of the encoder in the speech recognition model. Multi-level encoding features are generated based on the multiple single-level encoding features, and the output of the last encoding output layer is used as the bottleneck feature. In other words, the bottleneck feature is the feature corresponding to the last layer in the multi-level encoding feature.

[0096] At step S702, a content encoder is used to extract speech content information from the bottleneck feature. In some embodiments, the bottleneck feature includes speech content information in the speech. Based on this, the speech content information in the speech can be extracted from the bottleneck feature. Specifically, the content encoder can be used to extract speech content information from the bottleneck feature.

[0097] In step S703, the prosody information is extracted from the multi-level coding features using a trained prosody encoder. Since the prosody encoder can be trained based on the methods described in the above embodiments, it will not be described in detail here.

[0098] At step S704, the prosody information, speech content information, and target speaker label are input into a decoder to complete speech conversion. In some embodiments, the prosody information, speech content information, and target speaker label are fused to generate spectral features. Speech is then synthesized based on the spectral features to achieve speech conversion. Specifically, a vocoder can be used to recover the speech waveform from the spectral features to synthesize the speech.

[0099] The above-mentioned decoder can adopt the decoder structure in the prior art. The decoder can convert the read digital audio information into an analog audio signal output. For example, in the above-mentioned embodiment of the present disclosure, the multi-channel audio information encoded by the prosody encoder, content encoder and target speaker label is decoded and restored to realize the speech conversion process. In other words, the decoder can be used as a decompression structure. The hardware part of the decoder can be composed of an audio and video compression codec chip, a video gateway, a monitoring and management platform, input and output channels, a network interface, an audio and video interface, a serial interface control, a protocol interface control, and system software management. The software part includes a comformer structure similar to the aforementioned encoders.

[0100] Exemplary devices

[0101] After introducing the method of the exemplary embodiment of the present invention, next, reference is made to Figures 8 to 10 Related products of exemplary embodiments of the present invention are described.

[0102] Figure 8 The following is a schematic block diagram of a prosody encoder training device according to an embodiment of the present invention.

[0103] like Figure 8 As shown, the training device 800 includes a speech recognition module 801 and a prosody encoder 802. The speech recognition module 801 is configured to obtain multi-level coding features of the training data from speech training data. The multi-level coding features are extracted based on multiple coding output layers of the encoder of the speech recognition model. The prosody encoder 802 is connected to the speech recognition module 801 and is configured to perform training using the multi-level coding features of the training data, so that the trained prosody encoder can be used to extract prosody information.

[0104] Figure 9 The following schematically shows a block diagram of a speech conversion device according to an embodiment of the present invention.

[0105] like Figure 9As shown, the speech conversion device 900 includes a speech recognition module 901, a content encoder 902, a prosody encoder 903 and a decoder 904. The speech recognition module 901 is configured to obtain the bottleneck feature and multi-level coding feature of the speech to be converted from the speech. The multi-level coding feature is extracted based on multiple coding output layers of the encoder of the speech recognition model. The content encoder 902 is configured to extract speech content information from the bottleneck feature. The prosody encoder 903 is configured to extract prosody information from the multi-level coding feature. The prosody encoder can be trained based on the methods described in multiple embodiments above. The decoder 904 is connected to the speech recognition module 901, the content encoder 902 and the prosody encoder 903, and is configured to complete the speech conversion based on the prosody information, speech content information and the target speaker label.

[0106] Figure 10 A schematic block diagram of a device according to an embodiment of the present invention is schematically shown.

[0107] like Figure 10 As shown, the device 1000 may include a processor 1001 and a memory 1002. The memory 1002 stores computer instructions for training a prosody encoder. When the computer instructions are executed by the processor, the device performs the above and below combined Figure 2 and Figure 3 Furthermore, the memory also stores computer instructions for speech conversion, which, when executed by the processor, enable the device to perform the training of the prosody encoder described above. Figure 7 The speech conversion method described in .

[0108] In some embodiments, the device 1000 may perform training on a prosody encoder and use the prosody encoder to perform speech conversion, etc. Based on this, the device 1000 may obtain rich prosody information in the converted speech, thereby improving the effect of speech conversion.

[0109] In some implementation scenarios, the above-mentioned device 1000 can also be divided into multiple unit modules or devices, and each unit module is given a corresponding function. The solution of the present invention does not limit the structural design that the device 1000 can have.

[0110] It should be noted that although several devices or sub-devices of the device 1000 are mentioned in the above detailed description, this division is not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more devices described above may be embodied in a single device. Conversely, the features and functions of a single device described above may be further divided and embodied by multiple devices.

[0111] Furthermore, although the operations of the present method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in that particular order, or that all of the operations shown must be performed to achieve the desired results. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.

[0112] The use of the verbs "comprise", "include" and their conjugations in the application documents does not exclude the presence of elements or steps other than those stated in the application documents. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements.

[0113] Although the spirit and principles of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the specific embodiments disclosed, and the division into various aspects does not mean that the features of these aspects cannot be combined to benefit. Such division is merely for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A method for training a prosody encoder, characterized in that: include: Obtaining multi-level coding features of speech training data from the speech training data based on a speech recognition model; the multi-level coding features are extracted based on multiple coding output layers of an encoder of the speech recognition model; the coding features of each layer contain different proportions of speaker timbre, prosody, and language content information; Using the multi-level coding features of the training data to train the prosody encoder, so that the trained prosody encoder is used for extracting prosody information; The prosody encoder integrates a channel attention mechanism and a spatial attention mechanism to perform channel attention weighting and spatial attention weighting on the multi-level coding features to extract prosody features from the multi-level coding features.

2. The training method according to claim 1, characterized in that The speech recognition model includes an encoder having multiple encoding output layers, each encoding output layer is used to output a single-level encoding feature, and the multi-level encoding feature of the training data obtained from the training data based on the speech recognition model includes: Inputting the Mel spectrum of the training data into a speech recognition model; Extracting multiple single-level encoding features from multiple encoding output layers of an encoder in the speech recognition model; A multi-level coding feature is generated based on the multiple single-level coding features.

3. The training method according to claim 1, characterized in that The prosody encoder includes a channel attention module and a spatial attention module, wherein performing channel attention weighting and spatial attention weighting on the multi-level coding features includes: Utilizing the channel attention module to calculate the channel feature weight distribution corresponding to the multi-level encoding features of the training data to perform channel attention weighting; The spatial attention module is used to calculate the spatial feature weight distribution corresponding to the multi-level coding features after channel attention weighting to perform spatial attention weighting.

4. The training method according to claim 3, characterized in that The calculating the channel feature weight distribution corresponding to the multi-level coding features of the training data by using the channel attention module to perform channel attention weighting includes: The channel feature weights are multiplied by the multi-level coding features to obtain the multi-level coding features for updating the channel weights.

5. The training method according to claim 4, characterized in that The method of calculating the spatial feature weight distribution corresponding to the multi-level coding features after channel attention weighting by using the spatial attention module to perform spatial attention weighting includes: The spatial feature weights are multiplied by the multi-level coding features of the updated channel weights to obtain the coding information output by the prosody encoder.

6. The training method according to claim 1, characterized in that: The training of the prosody encoder using the multi-level coding features of the training data comprises: The prosody encoder obtains encoding information output by the prosody encoder according to the multi-level encoding features of the training data; The prosody encoder is subjected to gradient reversal adversarial training to remove residual speaker information in the encoded information output by the prosody encoder, thereby extracting the prosody information of the training data.

7. The training method according to claim 6, characterized in that The speech training data includes speaker labels, wherein adversarial training includes: Training a speaker classifier using the encoding information output by the prosody encoder and the speaker labels included in the training data to obtain a classification result; The weights of the neural network in the prosody encoder are reversely updated based on the classification result to complete gradient reversal, thereby removing residual speaker information in the encoded information output by the prosody encoder.

8. A prosody encoder training device, characterized in that: include: A speech recognition module configured to obtain multi-level coding features of speech training data from the training data; the multi-level coding features are extracted based on multiple coding output layers of an encoder of a speech recognition model; the coding features of each layer contain different proportions of speaker timbre, prosody, and language content information; a prosody encoder connected to the speech recognition module and configured to perform training using the multi-level coding features of the training data, so as to use the trained prosody encoder for extracting prosody information; The prosody encoder integrates a channel attention mechanism and a spatial attention mechanism to perform channel attention weighting and spatial attention weighting on the multi-level coding features to extract prosody features from the multi-level coding features.

9. A voice conversion method, characterized in that: include: Obtaining bottleneck features and multi-level coding features of speech from the speech to be converted based on a speech recognition model, wherein the multi-level coding features are extracted based on multiple coding output layers of an encoder of the speech recognition model; extracting speech content information from the bottleneck features using a content encoder; Extracting prosody information from the multi-level coding features using a trained prosody encoder, wherein the trained prosody encoder is trained based on the method according to any one of claims 1 to 7; as well as The prosody information, speech content information, and target speaker label are input into the decoder to complete the speech conversion.

10. The voice conversion method according to claim 9, characterized in that: The speech recognition model includes an encoder having multiple encoding output layers, each encoding output layer is used to output a single-level encoding feature, and the bottleneck feature and multi-level encoding feature of the speech to be converted are obtained from the speech to be converted based on the speech recognition model. Inputting the Mel spectrum of the speech to be converted into a speech recognition model; Extracting multiple single-level encoding features from multiple encoding output layers of an encoder in the speech recognition model; A multi-level coding feature is generated based on the multiple single-level coding features, and the output of the last coding output layer is used as the bottleneck feature.

11. The voice conversion method according to claim 9, wherein: Inputting prosody information, speech content information, and target speaker label into the decoder to complete speech conversion includes: fusing the prosody information, speech content information, and target speaker label to decode and generate spectral features; Speech is synthesized according to the spectral features to achieve speech conversion.

12. The voice conversion method according to claim 11, characterized in that: The synthesizing speech according to the spectral features to achieve speech conversion includes: A vocoder is used to recover the speech waveform from the spectral features to synthesize speech.

13. A voice conversion device, characterized in that: include: A speech recognition module configured to obtain a bottleneck feature and a multi-level coding feature of the speech from the speech to be converted, wherein the multi-level coding feature is extracted based on multiple coding output layers of an encoder of a speech recognition model; a content encoder configured to extract speech content information from the bottleneck features; A prosody encoder configured to extract prosody information from multi-level coding features, wherein the prosody encoder is trained based on the method according to any one of claims 1 to 7; as well as A decoder is connected to the speech recognition module, the content encoder and the prosody encoder, and is configured to complete speech conversion according to the prosody information, the speech content information and the target speaker label.

14. A device, characterized in that: include: processor; as well as A memory storing computer instructions for training a prosody encoder, wherein when the computer instructions are executed by the processor, the device executes the prosody encoder training method according to any one of claims 1 to 7; the memory further storing computer instructions for speech conversion, wherein when the computer instructions are executed by the processor, the device executes the speech conversion method according to any one of claims 9 to 12.

15. A computer-readable storage medium, characterized in that Contains program instructions for training a prosody encoder or performing speech conversion, and when the program instructions are executed by a processor, implements the prosody encoder training method according to any one of claims 1 to 7, or when the program instructions are executed by a processor, implements the speech conversion method according to any one of claims 9 to 12.

Images