Tone conversion methods, systems, storage media and electronic devices

By fusing and processing features from multiple models, the problems of unrealistic timbre conversion and limited range were solved, and high-fidelity timbre conversion was achieved.

CN117351973BActive Publication Date: 2026-11-10SHANGHAI MIDU INFORMATION TECH CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202311273422.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-09-27
Publication Date
2026-11-10
Estimated Expiration
2043-09-27

AI Technical Summary

Technical Problem

Existing timbre conversion methods suffer from problems such as unrealistic conversion results, timbre loss, or unclear content, and can only accept a limited number of timbre voices.

Method used

By employing multiple encoder modules and methods such as calculating mean and variance, and using Conformer, CTC, Bert, VQ-VAE, AE, and ResNet models, speech and text features are extracted and fused, and then inversely normalized and decoded to obtain high-fidelity converted speech.

Benefits of technology

It enables speech conversion based on different timbre characteristics, expands the range of timbre conversion, and effectively improves the fidelity and intelligence of speech.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117351973B_ABST
    Figure CN117351973B_ABST
Patent Text Reader

Abstract

This invention provides a timbre conversion method, system, storage medium, and electronic device, comprising: acquiring content speech and timbre speech; extracting speech recognition features and text features from the content speech; extracting normalized content features from the content speech; extracting a first timbre feature and a second timbre feature from the timbre speech; obtaining a fusion feature based on the speech recognition features, the text features, the normalized content features, the first timbre feature, and the second timbre feature; performing inverse normalization on the fusion feature to obtain an inverse normalized fusion feature; decoding the inverse normalized fusion feature to obtain reconstructed speech; decoding the fusion feature to obtain supplementary speech; and obtaining converted speech based on the reconstructed speech and the supplementary speech. The timbre conversion method, system, storage medium, and electronic device of this invention can achieve speech conversion with different timbre characteristics, effectively improving speech fidelity.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the technical field of speech processing, and in particular relates to a timbre conversion method, system, storage medium and electronic device. Background Technology

[0002] In existing technologies, timbre conversion typically includes the following steps:

[0003] 1) Receive two inputs: one is the speech to be converted, where the content of the speech is preserved; the other is the speech with the voice timbre, where the characteristics of the speech are preserved, such as male, female, or angry voice, etc.

[0004] 2) Output the voice with the completed timbre conversion.

[0005] However, the above-mentioned timbre conversion methods usually have the following problems:

[0006] 1) The conversion effect is often not realistic enough, either the timbre is lost or the content is unclear;

[0007] 2) It can only receive a limited number of voice timbres. For example, if a male voice is used as the voice timbre during training, then the voice output during inference can only contain a male voice.

[0008] To address these issues, this paper proposes a method that utilizes multiple encoder modules, patching, and mean-variance calculations to improve speech accuracy while enabling the conversion of speech with different timbre characteristics. Summary of the Invention

[0009] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a timbre conversion method, system, storage medium and electronic device that can realize speech conversion with different timbre characteristics and effectively improve the fidelity of speech.

[0010] In a first aspect, the present invention provides a timbre conversion method, the method comprising the following steps: acquiring content speech and timbre speech; extracting speech recognition features and text features from the content speech; extracting normalized content features from the content speech; extracting a first timbre feature and a second timbre feature from the timbre speech; acquiring a fusion feature based on the speech recognition features, the text features, the normalized content features, the first timbre feature, and the second timbre feature; performing inverse normalization on the fusion feature to acquire an inverse normalized fusion feature; decoding the inverse normalized fusion feature to acquire reconstructed speech; decoding the fusion feature to acquire supplementary speech; and acquiring converted speech containing content and timbre based on the reconstructed speech and the supplementary speech.

[0011] In one implementation of the first aspect, extracting the speech recognition features and text features of the content speech includes the following steps:

[0012] The encoder based on the Conformer model obtains the speech recognition features of the content speech;

[0013] The decoder based on the CTC model decodes the speech recognition features to obtain the speech recognition text;

[0014] The text features of the speech recognition text are obtained based on the BERT model.

[0015] In one implementation of the first aspect, extracting the normalized content features of the content speech includes the following steps:

[0016] The encoder based on the VQ-VAE model obtains the content features of the spoken content;

[0017] Calculate the mean and variance of the content features;

[0018] The content features are normalized based on the mean and variance to obtain the normalized content features.

[0019] In one implementation of the first aspect, extracting the first and second timbre features of the timbre speech includes the following steps:

[0020] The encoder based on the AE model obtains the first timbre feature of the timbre speech;

[0021] An encoder based on the ResNet model obtains the second timbre features of the timbre speech.

[0022] In one implementation of the first aspect, obtaining fusion features based on the speech recognition features, the text features, the normalized content features, the first timbre feature, and the second timbre feature includes the following steps:

[0023] Calculate the weighted sum of the fused features by combining the speech recognition features, the text features, the normalized content features, the first timbre feature, and the second timbre feature;

[0024] The weighted sum is processed using a self-attention mechanism to obtain the fusion features.

[0025] In one implementation of the first aspect, inverse normalization of the fused features to obtain inverse normalized fused features includes the following steps:

[0026] Obtain the mean and variance of the content features;

[0027] The fused features are inversely normalized based on the mean and variance to obtain the inversely normalized fused features.

[0028] In one implementation of the first aspect, decoding the inverse normalized fusion features to obtain reconstructed speech; decoding the fusion features to obtain supplementary speech includes the following steps:

[0029] The decoder based on the AE model decodes the inverse normalized fusion features;

[0030] The decoder based on the VQ-VAE model decodes the fused features.

[0031] Secondly, the present invention provides a timbre conversion system, the system comprising an acquisition module, a first extraction module, a second extraction module, a third extraction module, a fusion module, an inverse normalization module, a decoding module, and a conversion module;

[0032] The acquisition module is used to acquire content speech and timbre speech;

[0033] The first extraction module is used to extract the speech recognition features and text features of the content speech;

[0034] The second extraction module is used to extract the normalized content features of the content speech;

[0035] The third extraction module is used to extract the first timbre feature and the second timbre feature of the timbre speech;

[0036] The fusion module is used to obtain fusion features based on the speech recognition features, the text features, the normalized content features, the first timbre feature, and the second timbre feature;

[0037] The inverse normalization module is used to perform inverse normalization on the fused features to obtain inverse normalized fused features.

[0038] The decoding module is used to decode the inverse normalization fusion features to obtain reconstructed speech; and to decode the fusion features to obtain supplementary speech.

[0039] The conversion module is used to obtain converted speech containing content and timbre based on the reconstructed speech and the supplementary speech.

[0040] Thirdly, the present invention provides an electronic device, the electronic device comprising: a processor and a memory;

[0041] The memory is used to store computer programs;

[0042] The processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-described tone conversion method.

[0043] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by an electronic device, implements the above-described timbre conversion method.

[0044] As described above, the timbre conversion method, system, storage medium, and electronic device of the present invention have the following beneficial effects:

[0045] (1) It can realize speech conversion with different timbre characteristics, thus expanding the range of timbre conversion;

[0046] (2) Enhancement based on speech and text features of speech recognition effectively improves the fidelity of speech.

[0047] (3) It has a high degree of intelligence and is extremely practical. Attached Figure Description

[0048] Figure 1 The diagram shows a scenario of the electronic device of the present invention in one embodiment;

[0049] Figure 2 The flowchart shown is an embodiment of the timbre conversion method of the present invention;

[0050] Figure 3 The diagram shown is a schematic representation of the extraction of timbre features according to an embodiment of the present invention;

[0051] Figure 4 The diagram shown is a structural schematic of the timbre conversion system of the present invention in one embodiment.

[0052] Figure 5 The diagram shown is a structural schematic of an embodiment of the electronic device of the present invention. Detailed Implementation

[0053] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0054] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0055] The following embodiments of the present invention provide a timbre conversion method, which can be applied to, for example... Figure 1 The electronic devices shown are as follows. The electronic devices described in this invention may include mobile phones 11 with wireless charging capabilities, tablet computers 12, laptop computers 13, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. This invention does not impose any restrictions on the specific type of electronic device.

[0056] For example, the electronic device may be a station (STAION, ST) in a WLAN with wireless charging capability, a cellular phone, cordless phone, Session Initiation Protocol (SIP) phone, Wireless Local Loop (WLL) station, Personal Digital Assistant (PDA) device, handheld device with wireless charging capability, computing device or other processing device, computer, laptop computer, handheld communication device, handheld computing device, and / or other devices for communication over a wireless system, as well as next-generation communication systems, such as mobile terminals in 5G networks, mobile terminals in future evolved Public Land Mobile Networks (PLMNs), or mobile terminals in future evolved Non-terrestrial Networks (NTNs).

[0057] For example, the electronic device can communicate with networks and other devices wirelessly. The wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The GNSS can include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0058] The technical solutions of the present invention will now be described in detail with reference to the accompanying drawings.

[0059] like Figure 2 As shown, in one embodiment, the timbre conversion method of the present invention includes steps S1-S8.

[0060] Step S1: Obtain the content speech and the tone speech.

[0061] Specifically, when performing voice conversion, two inputs are required: the content speech and the voice tone. The content speech retains the content of the speech; the voice tone retains the characteristics of the speech, such as male, female, or angry voice.

[0062] Step S2: Extract the speech recognition features and text features of the content speech.

[0063] Specifically, the speech recognition features and text features are used to preserve the content features of the speech to the greatest extent possible.

[0064] In one embodiment, extracting the speech recognition features and text features of the content speech includes the following steps:

[0065] 21) The encoder based on the Conformer model obtains the speech recognition features of the content speech.

[0066] The Conformer model is a speech recognition model that is an improvement on the Transformer. The main improvement is that the Transformer is more effective at extracting long sequence dependencies, while convolution is better at extracting local features. Therefore, convolution is applied to the Encoder layer of the Transformer to improve the model’s performance on both long sequences and local features.

[0067] The speech recognition matrix is ​​a feature matrix with shape (sequence_len, dimension). `sequence_len` is the length of the speech downsampling, which is positively correlated with the length of the original speech input. The longer the speech, the longer the `sequence_len`. The longer the length of len, the greater the embedding dimension, which is 512.

[0068] 22) The decoder based on the CTC model decodes the speech recognition features to obtain the speech recognition text.

[0069] Among them, the CTC (Connectionist Temporal Classification) model is a way to avoid manual alignment of input and output, and can be understood as a time-series classification based on neural networks.

[0070] 23) Obtain the text features of the speech recognition text based on the BERT model.

[0071] The text feature is a feature vector with 768 elements.

[0072] Step S3: Extract the normalized content features of the spoken content.

[0073] Specifically, extracting normalized content features from the spoken content includes the following steps:

[0074] 31) The encoder based on the VQ-VAE model obtains the content features of the content speech.

[0075] VQ-VAE (Vector Quantized-Variational Autoencoder) is a generative model. A VAE model consists of three parts: the posterior distribution, the prior distribution, and the likelihood. The posterior distribution is learned using an encoder network, the likelihood is learned using a decoder network, and the prior distribution adopts a standard normal distribution with fixed parameters. During the VAE learning process, the posterior distribution is often assumed to be a diagonally variably normal multivariate distribution, while the latent variable is a continuous random variable.

[0076] 32) Calculate the mean and variance of the content features.

[0077] 33) Normalize the content features based on the mean and the variance to obtain the normalized content features.

[0078] The purpose of normalization is to minimize the inclusion of the timbre characteristics in the extracted features, thereby preserving the content features to the greatest extent possible. Preferably, the normalized content features are obtained according to (feature_content - mean) / var, where feature_content represents the content features, mean represents the mean, and var represents the variance.

[0079] Step S4: Extract the first timbre feature and the second timbre feature of the timbre speech.

[0080] Specifically, such as Figure 3 As shown, the encoder based on the AE model obtains the first timbre feature feature_speaker1 of the timbre speech; the encoder based on the ResNet model obtains the second timbre feature feature_speaker2 of the timbre speech, thereby using different encoders to extract different features of the timbre speech.

[0081] Step S5: Obtain fusion features based on the speech recognition features, the text features, the normalized content features, the first timbre feature, and the second timbre feature.

[0082] Specifically, firstly, a weighted sum of the fused features is calculated from the speech recognition features, the text features, the normalized content features, the first timbre feature, and the second timbre feature. In one embodiment, the weighted sum is calculated using the formula: feature_content_norm*0.4 + feature_speaker1*0.2 + feature_speaker2*0.2 + feature_text*0.1 + feature_asr*0.1, where feature_content_norm represents the normalized content features, feature_speaker1 represents the first timbre feature, feature_speaker2 represents the second timbre feature, feature_text represents the text features, and feature_asr represents the speech recognition features. Then, the weighted sum is processed using a self-attention mechanism to obtain the fused features.

[0083] Step S6: Perform inverse normalization on the fusion features to obtain inverse normalized fusion features.

[0084] Specifically, the inverse normalized fusion feature is obtained according to the formula feature_fusion*var+mean, where feature_fusion represents the fusion feature, var represents the variance of the content feature, and mean represents the mean of the content feature.

[0085] Step S7: Decode the inverse normalization fusion features to obtain reconstructed speech; decode the fusion features to obtain supplementary speech.

[0086] Specifically, the decoder based on the AE model decodes the inverse normalized fusion features to obtain the reconstructed speech; the decoder based on the VQ-VAE model decodes the fusion features to obtain supplementary speech, thus compensating for the low fidelity of the reconstructed speech. Here, AE (Auto-Encoder) is an unsupervised learning model. It uses the input data itself as supervision, based on the backpropagation algorithm and optimization methods (such as gradient descent), to guide the neural network to attempt to learn a mapping relationship, thereby obtaining a reconstructed output.

[0087] Step S8: Based on the reconstructed speech and the supplementary speech, obtain the converted speech containing content and timbre.

[0088] Specifically, the reconstructed speech and the supplemented speech are used as the converted speech. Preferably, the converted speech is calculated according to the formula 0.8*audio_new+0.2*audio_patcher, where audio_new represents the reconstructed speech and audio_patcher represents the supplemented speech.

[0089] The scope of protection for the timbre conversion method described in this embodiment is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this invention is included within the scope of protection of this invention.

[0090] This invention also provides a timbre conversion system, which can implement the timbre conversion method described in this invention. However, the implementation device of the timbre conversion system described in this invention includes, but is not limited to, the structure of the timbre conversion system listed in this embodiment. All structural modifications and substitutions of the prior art made according to the principles of this invention are included within the protection scope of this invention.

[0091] like Figure 4 As shown, in one embodiment, the timbre conversion system of the present invention includes an acquisition module 41, a first extraction module 42, a second extraction module 43, a third extraction module 44, a fusion module 45, an inverse normalization module 46, a decoding module 47, and a conversion module 48.

[0092] The acquisition module 41 is used to acquire the content speech and the tone speech.

[0093] The first extraction module 42 is connected to the acquisition module 41 and is used to extract the speech recognition features and text features of the content speech.

[0094] The second extraction module 43 is connected to the acquisition module 41 and is used to extract the normalized content features of the content speech.

[0095] The third extraction module 44 is connected to the acquisition module 41 and is used to extract the first timbre feature and the second timbre feature of the timbre speech.

[0096] The fusion module 45 is connected to the first extraction module 42, the second extraction module 43 and the third extraction module 44, and is used to obtain fusion features based on the speech recognition features, the text features, the normalized content features, the first timbre features and the second timbre features.

[0097] The inverse normalization module 46 is connected to the fusion module 45 and is used to perform inverse normalization on the fusion features to obtain inverse normalized fusion features.

[0098] The decoding module 47 is connected to the fusion module 45 and the inverse normalization module 46, and is used to decode the inverse normalization fusion features to obtain reconstructed speech; and decode the fusion features to obtain supplementary speech.

[0099] The conversion module 48 is connected to the decoding module 47 and is used to obtain converted speech containing content and timbre based on the reconstructed speech and the supplementary speech.

[0100] The structure and principle of the acquisition module 41, the first extraction module 42, the second extraction module 43, the third extraction module 44, the fusion module 45, the inverse normalization module 46, the decoding module 47, and the conversion module 48 correspond one-to-one with the steps in the above-mentioned timbre conversion method, so they will not be described in detail here.

[0101] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0102] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of the present invention, depending on actual needs. For example, the functional modules / units in the various embodiments of the present invention may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0103] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0104] This invention also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. This available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0105] This invention also provides an electronic device. The electronic device includes a processor and a memory.

[0106] The memory is used to store computer programs.

[0107] The memory includes various media capable of storing program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card, or optical disk.

[0108] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the electronic device performs the above-described timbre conversion method.

[0109] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0110] like Figure 5As shown, the electronic device of the present invention is embodied in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors or processing units 51, a memory 52, and a bus 53 connecting different system components (including the memory 52 and the processing unit 51).

[0111] Bus 53 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0112] Electronic devices typically include a variety of computer-readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, and removable and non-removable media.

[0113] Memory 52 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 521 and / or cache memory 522. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 523 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 5 Not shown; usually referred to as a "hard drive"). Although Figure 5 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 53 via one or more data media interfaces. Memory 52 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0114] A program / utility 524 having a set (at least one) of program modules 5241 may be stored, for example, in memory 52. ​​Such program modules 5241 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 5241 typically perform the functions and / or methods described in the embodiments of the present invention.

[0115] The electronic device can also communicate with one or more external devices (e.g., keyboard, pointing device, display, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., network interface card, modem, etc.). This communication can be performed through input / output (I / O) interface 54. Furthermore, the electronic device can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 55. Figure 5 As shown, network adapter 55 communicates with other modules of the electronic device via bus 53. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0116] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A timbre conversion method, characterized in that, The method includes the following steps: Get the content's voice and the voice's tone; Extract the speech recognition features and text features of the content speech; Extract normalized content features from the spoken content; Extract the first and second timbre features of the timbre speech; obtain the first and second timbre features based on two different encoders; A fusion feature is obtained based on the speech recognition feature, the text feature, the normalized content feature, the first timbre feature, and the second timbre feature; The fusion features are inversely normalized to obtain inversely normalized fusion features; The inverse normalized fusion features are decoded to obtain reconstructed speech; the fusion features are decoded to obtain supplementary speech. Based on the reconstructed speech and the supplementary speech, a converted speech containing content and timbre is obtained; Extracting normalized content features from the spoken content includes the following steps: The encoder based on the VQ-VAE model obtains the content features of the spoken content; Calculate the mean and variance of the content features; The content features are normalized based on the mean and the variance to obtain the normalized content features. Obtaining fused features based on the speech recognition features, the text features, the normalized content features, the first timbre feature, and the second timbre feature includes the following steps: Calculate the weighted sum of the fused features by taking the speech recognition features, the text features, the normalized content features, the first timbre feature, and the second timbre feature; The weighted sum is processed using a self-attention mechanism to obtain the fusion features; The inverse normalized fusion feature is obtained according to the formula feature_fusion*var+mean, where feature_fusion represents the fusion feature, var represents the variance of the content feature, and mean represents the mean of the content feature.

2. The timbre conversion method according to claim 1, characterized in that: Extracting the speech recognition features and text features of the content includes the following steps: The encoder based on the Conformer model obtains the speech recognition features of the content speech; The decoder based on the CTC model decodes the speech recognition features to obtain the speech recognition text; The text features of the speech recognition text are obtained based on the BERT model.

3. The timbre conversion method according to claim 1, characterized in that: Extracting the first and second timbre features of the timbre speech includes the following steps: The encoder based on the AE model obtains the first timbre feature of the timbre speech; An encoder based on the ResNet model obtains the second timbre features of the timbre speech.

4. The timbre conversion method according to claim 1, characterized in that: The process of inversely normalizing the fused features to obtain inversely normalized fused features includes the following steps: Obtain the mean and variance of the content features; The fused features are inversely normalized based on the mean and variance to obtain the inversely normalized fused features.

5. The timbre conversion method according to claim 1, characterized in that: Decoding the inverse normalized fusion features to obtain reconstructed speech; decoding the fusion features to obtain supplementary speech includes the following steps: The decoder based on the AE model decodes the inverse normalized fusion features; The decoder based on the VQ-VAE model decodes the fused features.

6. A timbre conversion system, characterized in that, The system includes an acquisition module, a first extraction module, a second extraction module, a third extraction module, a fusion module, an inverse normalization module, a decoding module, and a conversion module; The acquisition module is used to acquire content speech and timbre speech; The first extraction module is used to extract the speech recognition features and text features of the content speech; The second extraction module is used to extract the normalized content features of the content speech; The third extraction module is used to extract the first timbre feature and the second timbre feature of the timbre speech; The first timbre feature and the second timbre feature are obtained based on two different encoders; The fusion module is used to obtain fusion features based on the speech recognition features, the text features, the normalized content features, the first timbre feature, and the second timbre feature; The inverse normalization module is used to perform inverse normalization on the fused features to obtain inverse normalized fused features. The decoding module is used to decode the inverse normalization fusion features to obtain reconstructed speech; and to decode the fusion features to obtain supplementary speech. The conversion module is used to obtain converted speech containing content and timbre based on the reconstructed speech and the supplementary speech; Extracting normalized content features from the spoken content includes the following steps: The encoder based on the VQ-VAE model obtains the content features of the spoken content; Calculate the mean and variance of the content features; The content features are normalized based on the mean and the variance to obtain the normalized content features. Obtaining fused features based on the speech recognition features, the text features, the normalized content features, the first timbre feature, and the second timbre feature includes the following steps: Calculate the weighted sum of the fused features by taking the speech recognition features, the text features, the normalized content features, the first timbre feature, and the second timbre feature; The weighted sum is processed using a self-attention mechanism to obtain the fusion features; The inverse normalized fusion feature is obtained according to the formula feature_fusion*var+mean, where feature_fusion represents the fusion feature, var represents the variance of the content feature, and mean represents the mean of the content feature.

7. An electronic device, characterized in that, The electronic device includes: a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory to cause the electronic device to perform the timbre conversion method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by an electronic device, it implements the timbre conversion method as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • Voice conversion model training method and device, electronic equipment and medium

    CN113689868A