A voice conversion method, device, equipment and computer readable storage medium

Abstract

The application discloses a speech conversion method and device, equipment and a computer readable storage medium, based on a conditional VAE framework, using an encoder to extract the timbre information and content information of the speech respectively, and then using a generator to synthesize the speech, and in order to decouple the speech into content information and timbre information, a pre-trained self-supervised speech representation learning model is introduced to improve the content decoupling capability, and a feature vector x containing source content information C output by the self-supervised speech representation learning model is input into a flow model to obtain a feature vector x ssl As a conditional constraint content information C decoupling, using the forward and reverse characteristics of the flow model, using two KL divergence criteria to guide the model parameter connection, ensuring the lossless of the content information, and modeling the timbre information as a standard normal distribution to reduce the timbre leakage problem, realizing the improvement of the decoupling capability of the content information, and solving the technical problem of timbre leakage.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of voice conversion, and in particular to a voice conversion method, device, equipment and computer readable storage medium. BACKGROUND

[0002] Voice conversion technology, also known as voice transformation technology, is a fascinating and widely applicable technology in the field of speech processing. It involves converting the voice of one person into the voice of another person while maintaining the integrity of the original language content.

[0003] Voice conversion technology exhibits its importance in multiple fields, such as film and television post-production, speech synthesis, privacy protection, language learning, audio editing, and assisting the disabled. It can enhance user experience, enhance privacy security, and provide technical support for the creative industry.

[0004] Traditional voice conversion methods rely on statistical models and require the collection of parallel voice data sets of source speakers and target speakers. These data are used to train a conversion module that can learn the acoustic feature mapping relationship between source and target speakers.

[0005] With the development of deep learning technology, the field of voice conversion has undergone significant technological innovation. Modern voice conversion models use deep neural networks to more effectively capture and convert the features of voice signals.

[0006] Existing research has begun to explore end-to-end voice conversion models, which can directly convert from source voice to target voice without explicitly modeling intermediate features.

[0007] A key direction of modern voice conversion technology is to decouple the content information and timbre information in the voice. This allows the model to process the semantic and sound features of the voice separately, making voice conversion more flexible. Timbre information is usually represented by a voiceprint vector (such as i-vector or d-vector), while content information can be represented by phonetic posterior-grams or other acoustic features.

[0008] How to improve the decoupling ability of content information while reducing the problem of timbre leakage is an urgent problem for those skilled in the art. SUMMARY

[0009] The present application provides a voice conversion method, device, equipment and computer readable storage medium, which realizes the improvement of the decoupling ability of content information and solves the technical problem of timbre leakage.

[0010] Therefore, the first aspect of the present application provides a voice conversion method, which comprises:

[0011] Step S1, obtaining source speaker voice and target speaker voice;

[0012] Step S2, inputting the source speaker voice into a pre-trained self-supervised speech representation learning model to obtain a feature vector x containing source content information C ssl ;

[0013] Step S3, inputting the feature vector x ssl as a decoupled input of the source content information C into a flow model, and training by using a KL divergence criterion to obtain target content information;

[0014] Step S4, encoding the source speaker voice and the target speaker voice by using an encoder to obtain first vocal information of the source speaker and second vocal information of the target speaker, respectively;

[0015] Step S5, replacing the first vocal information of the source speaker with the second vocal information of the target speaker, and reconstructing the second vocal information and the target content information by using a generator to generate target voice.

[0016] Optionally, the step S1 specifically comprises:

[0017] Respectively obtaining source speaker voice and target speaker voice, and performing denoising and normalization preprocessing on the source speaker voice and the target speaker voice.

[0018] Optionally, the step S2 specifically comprises:

[0019] Inputting the source speaker voice into a pre-trained self-supervised speech representation learning model as an input to obtain a feature vector x ssl of a target dimension containing source content information C, wherein the self-supervised speech representation learning model is specifically a HuBERT model.

[0020] Optionally, the step S3 specifically comprises:

[0021] Inputting the feature vector x ssl as a decoupled conditional input of the source content information C into a flow model;

[0022] In the flow model, the source content information C is decoupled and trained by using a KL divergence criterion to obtain target content information.

[0023] Optionally, the step S3 further comprises:

[0024] In the decoupled training process, the source content information C is transformed into a Gaussian distribution The mean and variance contained in the feature vector are calculated by a preset residual block network.

[0025] Optionally, the decoupling training specifically comprises:

[0026] In the forward process, the flow model converts the source content information C to ss under the constraint of the feature vector x

[0027] calculates the first KL divergence between the target Gaussian distribution and the target content information C;

[0028] In the reverse process, the flow model restores the training content information C by the inverse transformation f -1

[0029] calculates the second KL divergence between the training content information and the source content information C;

[0030] combines the first KL divergence and the second KL divergence into a loss function of the flow model, and iteratively trains the flow model until the loss function converges.

[0031] Optionally, the step S4 specifically comprises:

[0032] calculates a first linear spectrum and a second linear spectrum of the source speaker voice and the target speaker voice respectively;

[0033] inputs the first linear spectrum and the second linear spectrum into an encoder as input, and outputs first timbre information S = E s (x) of the source speaker and second timbre information

[0034] The second aspect of the application provides a speech conversion device, which comprises:

[0035] an acquisition unit configured to acquire a source speaker voice and a target speaker voice;

[0036] a first processing unit configured to input the source speaker voice into a pre-trained self-supervised speech representation learning model to obtain a feature vector x ssl containing source content information C;

[0037] a second processing unit configured to input the feature vector x ssl into a flow model as a decoupling input of the source content information C, and train the flow model by using a KL divergence criterion to obtain target content information;

[0038] ​​​a third processing unit, configured to encode the source speaker voice and the target speaker voice by an encoder to obtain first timbre information of the source speaker and second timbre information of the target speaker respectively;

[0039] a voice conversion unit, configured to replace the first timbre information of the source speaker with the second timbre information of the target speaker, and reconstruct the second timbre information and the target content information by a generator to generate target voice.

[0040] Optionally, the acquisition unit is specifically configured to:

[0041] acquire the source speaker voice and the target speaker voice respectively, and perform denoising and normalization preprocessing on the source speaker voice and the target speaker voice.

[0042] Optionally, the first processing unit is specifically configured to:

[0043] input the source speaker voice into a pre-trained self-supervised speech representation learning model to obtain a feature vector x of a target dimension containing source content information C ss l, wherein the self-supervised speech representation learning model is specifically a HuBERT model.

[0044] Optionally, the second processing unit is specifically configured to:

[0045] input the feature vector x ssl into a flow model as a constraint condition for decoupling the source content information C;

[0046] In the flow model, the source content information C is decoupled and trained by a KL divergence criterion to obtain target content information.

[0047] Optionally, the second processing unit is further configured to:

[0048] in the decoupling training process, transform the source content information C into a Gaussian distribution containing mean and variance calculated by a preset residual block network from the feature vector.

[0049] Optionally, the decoupling training is specifically:

[0050] in the forward process, the flow model converts the source content information C to ss under the constraint of the feature vector x

[0051] calculates a first KL divergence between and the target Gaussian distribution;

[0052] In the reverse process, the flow model is transformed by an inverse transformation f -1 The restored to the training content information;

[0053] calculating a second KL divergence between the training content information and the source content information C;

[0054] combining the first KL divergence and the second KL divergence as a loss function of the flow model, iteratively training the flow model until the loss function converges.

[0055] Optionally, the third processing unit is specifically configured to:

[0056] respectively calculating first linear spectra and second linear spectra of the source speaker voice and the target speaker voice;

[0057] taking the first linear spectra and the second linear spectra as inputs of an encoder, and respectively outputting first timbre information S = E s (x) of the source speaker and second timbre information

[0058] The third aspect of the present application provides a voice conversion device, the device comprising a processor and a memory:

[0059] The memory is configured to store program code and transmit the program code to the processor;

[0060] The processor is configured to execute the steps of the voice conversion method according to the instructions in the program code.

[0061] The fourth aspect of the present application provides a computer readable storage medium for storing program code, the program code being used to execute the method of the first aspect.

[0062] From the above technical solutions, it can be seen that the embodiments of the present application have the following advantages:

[0063] In the present application, a voice conversion method, device, equipment and computer readable storage medium are provided, based on a conditional VAE framework, using an encoder to extract timbre information and content information of a voice, and then using a generator to synthesize the voice, and in order to decouple the voice into content information and timbre information, a pre-trained self-supervised speech representation learning model is introduced to improve the content decoupling ability, and a flow model is used to transform the feature vector x sslAs a decoupling of the conditional constraint content information C, the forward and reverse characteristics of the flow model are used to guide the model parameter connection using two KL divergence criteria, ensure the lossless of the content information, model the timbre information as a standard normal distribution to reduce the timbre leakage problem, realize the decoupling ability of the content information, and solve the technical problem of timbre leakage. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 A method flowchart of the speech conversion method in the embodiments of the present application;

[0065] Figure 2 A structure schematic diagram of the speech conversion device in the embodiments of the present application;

[0066] Figure 3 A structure schematic diagram of the speech conversion device in the embodiments of the present application;

[0067] Figure 4 A conditional VAE framework structure diagram in the embodiments of the present application;

[0068] Figure 5 A residual block network structure diagram in the embodiments of the present application. DETAILED DESCRIPTION

[0069] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor are within the scope of protection of the present application.

[0070] The present application designs a speech conversion method, device, equipment and computer readable storage medium, based on the conditional VAE framework, using the encoder to extract the timbre information and content information of the speech respectively, and then synthesizing the speech through the generator, in order to decouple the speech into content information and timbre information, introducing the pre-trained hubert model to improve the content decoupling ability, using the flow model to obtain the feature vector x ssl As a decoupling of the conditional constraint content information C, the forward and reverse characteristics of the flow model are used to guide the model parameter connection using two KL divergence criteria, ensure the lossless of the content information, model the timbre information as a standard normal distribution to reduce the timbre leakage problem, realize the decoupling ability of the content information, and solve the technical problem of timbre leakage.

[0071] For easy understanding, please refer to Figure 1 , Figure 1For the method flowchart of the voice conversion method in the embodiments of the present application, as shown in Figure 1 , specifically:

[0072] Step S1, obtaining source speaker voice and target speaker voice;

[0073] It should be noted that first, the source speaker voice of the source speaker and the target speaker voice of the target speaker are collected respectively.

[0074] In order to ensure that the quality of the voice sample meets the requirements such as intelligibility and no obvious background noise, the source speaker voice and the target speaker voice can be further preprocessed such as denoising and normalization to improve the accuracy of subsequent processing.

[0075] The present application is based on the conditional VAE framework as shown in Figure 4 , which contains a content encoder E and a generator G. Assuming that the voice x is composed of content information C and timbre S, the content information C describes the language content between different speakers, and the timbre S describes the timbre between different speakers, that is, C=E c (x), S=E s (x).

[0076] Step S2, inputting the source speaker voice into the pre-trained self-supervised voice representation learning model to obtain a feature vector x ssl containing source content information C;

[0077] It should be noted that the HuBERT model is a self-supervised voice representation learning model. The full name of HuBERT is "Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units", that is, self-supervised voice representation learning by masked prediction of hidden units.

[0078] HuBERT uses a Transformer-based encoder, which is a common component of many advanced voice processing models currently, and can process sequence data and capture long-distance dependencies.

[0079] During training, a certain proportion of voice frames are randomly selected and replaced with a special [MASK] mark, and the model needs to predict these masked frames.

[0080] Because of the use of self-supervised learning, HuBERT does not require a large amount of labeled data, which makes it very useful in data-scarce situations.

[0081] The feature representation learned by HuBERT has good generalization ability and can adapt to different voice processing tasks.

[0082] HuBERT provides an end-to-end framework that can directly learn useful feature representations from raw speech waveforms.

[0083] A 256-dimensional feature vector x containing source content information C is learned from the source speaker's speech using the HuberT model. ssl .

[0084] The feature vector x output by the Hubert model ssl Further optimization or selection may be needed to retain the most useful information.

[0085] Step S3: Convert the feature vector x ssl The decoupled source content information C, which serves as a condition constraint, is input into the flow model and trained using the KL divergence criterion to obtain the target content information.

[0086] It should be noted that, utilizing the reversible property of the flow model, relying on x ssl Decoupling of eigenvector constraint content information C.

[0087] The decoupling process needs to ensure the integrity and independence of the content information, which may involve hyperparameter adjustment and model fine-tuning.

[0088] The Flow model will include content information Transform to Gaussian distribution Right now Among them, mean and variance By x ssl The feature vectors are obtained through residual block computation. The residual block network structure is as follows: Figure 5 As shown.

[0089] To achieve x ssl The decoupling of the feature constraint content information C satisfies the similarity of the two Gaussian distributions mentioned above. Training is performed using the KL divergence criterion, as shown in the following formula:

[0090]

[0091] Taking full advantage of the reversible nature of flow, the KL divergence criterion also applies to the inverse process, as shown in the following formula:

[0092]

[0093] Step S4: Encode the source speaker's speech and the target speaker's speech using an encoder to obtain the first timbre information of the source speaker and the second timbre information of the target speaker, respectively.

[0094] It should be noted that, in order to model timbre information and reduce timbre leakage, timbre information... Assuming the distribution follows a standard normal distribution, the KL divergence criterion is also used for constraint. Approximate standard normal distribution:

[0095]

[0096] The recovery loss function is:

[0097]

[0098] Step S5: Replace the source speaker's first timbre information with the target speaker's second timbre information, and use a generator to reconstruct the second timbre information and target content information to generate the target speech.

[0099] Furthermore, step S1 specifically includes:

[0100] The source speaker's speech and the target speaker's speech are acquired separately, and the source speaker's speech and the target speaker's speech are preprocessed by denoising and normalization.

[0101] It should be noted that, in order to ensure that the quality of the speech samples meets the requirements, such as clarity and absence of obvious background noise, the source speaker's speech and the target speaker's speech can be further preprocessed, such as denoising and normalization, to improve the accuracy of subsequent processing.

[0102] Furthermore, step S3 specifically includes:

[0103] The feature vector x ssl The decoupling condition of the constraint source content information C is input into the flow model;

[0104] In the flow model, the source content information C is decoupled and trained using the KL divergence criterion to obtain the target content information.

[0105] Furthermore, step S3 also includes:

[0106] During the decoupled training process, the source content information C is transformed into a Gaussian distribution. The mean and variance contained therein are obtained by calculating the feature vectors through a preset residual block network.

[0107] It should be noted that the Flow model incorporates content information. Transform to Gaussian distribution Right now Among them, mean and variance By x ssl The feature vectors are obtained through residual block computation. The residual block network structure is as follows: Figure 5 As shown.

[0108] Furthermore, the decoupling training specifically involves:

[0109] In the forward process, the flow model converts the source content information C to the target content information C' through the transformation f under the constraint of the feature vector x ssl

[0110] Computes the first KL divergence between the target Gaussian distribution and the target content information C';

[0111] In the inverse process, the flow model restores the training content information C' through the inverse transformation f -1

[0112] Computes the second KL divergence between the training content information C and the source content information C;

[0113] Combines the first KL divergence and the second KL divergence as the loss function of the flow model, iteratively trains the flow model until the loss function converges.

[0114] Further, step S4 specifically comprises:

[0115] Respectively computes the first linear spectrum and the second linear spectrum of the source speaker voice and the target speaker voice;

[0116] Takes the first linear spectrum and the second linear spectrum as the input of the encoder, and outputs the first timbre information S = E s (x) of the source speaker and the second timbre information S' = E'(x') of the target speaker by the encoder respectively;

[0117] It should be noted that the fast Fourier transform (FFT) is performed on the source speaker voice and the target speaker voice to convert the time domain signal to the frequency domain.

[0118] The linear spectrum is extracted from the FFT result. The linear spectrum is a logarithmic representation of the power spectrum, and the power of each frequency point is usually represented using a 10 times logarithm.

[0119] The linear spectrum is filtered using a Mel-filterbank (Mel-filterbank) to simulate the perceptual sensitivity of the human ear to different frequencies.

[0120] The filtered signal is subjected to a discrete cosine transform, usually DCT-II type, to reduce the redundancy of the data.

[0121] Features are extracted from the DCT result, such as Mel-frequency cepstral coefficients (MFCCs) or other speech features.

[0122] The extracted features are taken as the input of the encoder. In a self-supervised learning model such as HuBERT, these features will be further processed to learn the representation of speech. ​​​​

[0123] Please refer to Figure 2 , Figure 2 The structure of the speech conversion device in the embodiment of the application is shown in FIG. 1, and specifically is as follows: Figure 2

[0124] The acquisition unit 201 is configured to acquire source speaker speech and target speaker speech.

[0125] It should be noted that first, the source speaker speech and the target speaker speech of the source speaker and the target speaker are respectively collected.

[0126] In order to ensure that the quality of the speech sample meets the requirements such as intelligibility and no obvious background noise, the source speaker speech and the target speaker speech can be further preprocessed such as denoising and normalization to improve the accuracy of subsequent processing.

[0127] The application is based on a conditional VAE framework as shown in FIG. 2, which includes a content encoder E and a generator G. It is assumed that the speech x is composed of content information C and timbre S, the content information C describes the language content between different speakers, and the timbre S describes the timbre between different speakers, that is, C=E c (x), S=E s (x). Figure 4

[0128] The first processing unit 202 is configured to input the source speaker speech into a pre-trained self-supervised speech representation learning model to obtain a feature vector x ssl containing source content information C.

[0129] It should be noted that the HuBERT model is a self-supervised speech representation learning model. The full name of HuBERT is "Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units", that is, self-supervised speech representation learning by masked prediction of hidden units.

[0130] HuBERT uses a Transformer-based encoder, which is a common component of many advanced speech processing models currently, and can process sequence data and capture long-distance dependencies.

[0131] During training, a certain proportion of speech frames are randomly selected and replaced with a special [MASK] mark, and the model needs to predict these masked frames.

[0132] Because of the use of self-supervised learning, HuBERT does not require a large amount of labeled data, which makes it very useful in data-scarce situations. ​​

[0133] The feature representations learned by HuBERT have good generalization ability and can be adapted to different speech processing tasks.

[0134] HuBERT provides an end-to-end framework that can learn useful feature representations directly from raw speech waveforms.

[0135] A 256-dimensional feature vector x containing source content information C is learned from the source speaker's speech using the HuberT model. ssl .

[0136] The feature vector x output by the Hubert model ssl Further optimization or selection may be needed to retain the most useful information.

[0137] The second processing unit 203 is used to process the feature vector x ssl The decoupled source content information C, which serves as a condition constraint, is input into the flow model and trained using the KL divergence criterion to obtain the target content information.

[0138] It should be noted that, utilizing the reversible property of the flow model, relying on x ssl Decoupling of eigenvector constraint content information C.

[0139] The decoupling process needs to ensure the integrity and independence of the content information, which may involve hyperparameter adjustment and model fine-tuning.

[0140] The Flow model will include content information Transform to Gaussian distribution Right now Among them, mean and variance By x ssl The feature vectors are obtained through residual block computation. The residual block network structure is as follows: Figure 5 As shown.

[0141] To achieve x ssl The decoupling of the feature constraint content information C satisfies both the similarity of the two Gaussian distributions mentioned above. Training is performed using the KL divergence criterion, as shown in the following formula:

[0142]

[0143] Taking full advantage of the reversible nature of flow, the KL divergence criterion also applies to the inverse process, as shown in the following formula:

[0144]

[0145] The third processing unit 204 is used to encode the source speaker's speech and the target speaker's speech through an encoder to obtain the first timbre information of the source speaker and the second timbre information of the target speaker, respectively.

[0146] It should be noted that, in order to model the timbre information to reduce the timbre leakage problem, the timbre information Assuming that it is subject to a standard normal distribution, the KL divergence criterion is also used to constrain Approximate standard normal distribution:

[0147]

[0148] The simultaneous recovery loss function is:

[0149]

[0150] The speech conversion unit 205 is configured to replace the first timbre information of the source speaker with the second timbre information of the target speaker, and reconstruct the second timbre information and the target content information by using the generator to generate the target speech.

[0151] Further, the acquisition unit 201 is specifically configured to:

[0152] The source speaker speech and the target speaker speech are acquired respectively, and the source speaker speech and the target speaker speech are preprocessed by denoising and normalization.

[0153] It should be noted that, in order to ensure that the quality of the speech sample meets the requirements, such as intelligibility, no obvious background noise, etc., the source speaker speech and the target speaker speech can be further preprocessed, such as denoising, normalization, etc., to improve the accuracy of subsequent processing.

[0154] Further, the second processing unit 203 is specifically configured to:

[0155] The feature vector x ssl is input into the flow model as a constraint condition for decoupling the source content information C;

[0156] In the flow model, the source content information C is decoupled and trained by using the KL divergence criterion to obtain the target content information.

[0157] Further, the second processing unit 203 is further configured to:

[0158] In the decoupling training process, the source content information C is transformed to a Gaussian distribution The mean and variance contained in the Gaussian distribution

[0159] It should be noted that, in the flow model, the content information is transformed to a Gaussian distribution that is where the mean and variance are calculated by x sslThe feature vector is calculated by a residual block, and the network structure of the residual block is as shown in Figure 5

[0160] Further, the decoupling training specifically includes:

[0161] In the forward process, the flow model converts the source content information C to ssl under the constraint of the feature vector x

[0162] calculates the first KL divergence between and the target Gaussian distribution;

[0163] In the reverse process, the flow model restores -1 to the training content information through the inverse transformation f

[0164] calculates the second KL divergence between the training content information and the source content information C;

[0165] combines the first KL divergence and the second KL divergence into a loss function of the flow model, and iteratively trains the flow model until the loss function converges.

[0166] Further, the third processing unit 204 is specifically configured to:

[0167] respectively calculate the first linear spectrum and the second linear spectrum of the source speaker voice and the target speaker voice;

[0168] take the first linear spectrum and the second linear spectrum as inputs of the encoder, and respectively output the first timbre information S=E s (x) of the source speaker and the second timbre information

[0169] It should be noted that the fast Fourier transform (FFT) is performed on the source speaker voice and the target speaker voice to convert the time domain signal to the frequency domain.

[0170] The linear spectrum is extracted from the FFT result. The linear spectrum is a logarithmic representation of the power spectrum, and the power of each frequency point is usually represented using a logarithm of 10.

[0171] The linear spectrum is filtered by applying a Mel-filterbank (Mel-filterbank) to simulate the perceptual sensitivity of the human ear to different frequencies.

[0172] The filtered signal is subjected to a discrete cosine transform, usually DCT-II type, to reduce the redundancy of the data.

[0173] Features such as Mel-frequency cepstral coefficients (MFCCs) or other speech features are extracted from the DCT result.​​

[0174] The extracted features are input to the encoder. In a self-supervised learning model such as HuBERT, these features are further processed to learn a representation of speech.

[0175] Embodiments of the present application also provide another speech conversion device, such as Figure 3 As shown, for ease of illustration, only parts related to the embodiments of the present application are shown, and specific technical details not disclosed are referred to the method part of the embodiments of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), a sales terminal (English full name: Point of Sales, English abbreviation: POS), a vehicle-mounted computer, etc. Take the terminal as a mobile phone for example:

[0176] Figure 3 The diagram shows the partial structure of the mobile phone related to the terminal provided by the embodiments of the present application. Referring to Figure 3 , the mobile phone includes: radio frequency (English full name: Radio Frequency, English abbreviation: RF) circuit 1010, memory 1020, input unit 1030, display unit 1040, sensor 1050, audio circuit 1060, wireless fidelity (English full name: wireless fidelity, English abbreviation: WiFi) module 1070, processor 1080, and power supply 1090, etc. Those skilled in the art can understand, Figure 3 that the structure of the mobile phone shown in the above does not constitute a limitation on the mobile phone, and can include more or less components than the diagram, or combine certain components, or different component arrangements.

[0177] The specific introduction of each component of the mobile phone is as follows: Figure 3

[0178] ​The RF circuit 1010 can be configured to receive and send signals during a call or data session, particularly, to receive downlink information from a base station and send uplink data to the base station. Typically, the RF circuit 1010 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 1010 can be configured to communicate with the network and other devices through wireless communication. The wireless communication can use any communication standards or protocols, including but not limited to Global System for Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0179] The memory 1020 can be configured to store software programs and modules, and the processor 1080 can be configured to execute various functions of the mobile phone and process data by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required for a function (such as a sound playing function, an image playing function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device.

[0180] The input unit 1030 can be used to receive inputted digital or character information, and to generate key signal input with respect to user setting of the mobile phone and function control. Specifically, the input unit 1030 can include a touch panel 1031 and other input devices 1032. The touch panel 1031, also called a touch screen, can collect a touch operation (such as an operation of a user using a finger, a stylus, or any suitable object or accessory on or near the touch panel 1031) of the user on or near the touch panel 1031, and drive a corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 can include two parts of a touch detection device and a touch controller. The touch detection device detects a touch position of the user and detects a signal caused by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch coordinates, and sends it to the processor 1080, and can receive a command from the processor 1080 and execute it. In addition, the touch panel 1031 can be implemented in various types such as a resistive type, a capacitive type, an infrared type, and a surface acoustic wave type. In addition to the touch panel 1031, the input unit 1030 can also include other input devices 1032. Specifically, the other input devices 1032 can include one or more of a physical keyboard, a function key (such as a volume control key, an on-off key, etc.), a trackball, a mouse, a joystick, etc.

[0181] The display unit 1040 can be used to display information input by the user or information provided to the user, and various menus of the mobile phone. The display unit 1040 can include a display panel 1041, which can be configured in the form of a liquid crystal display (English full name: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (English full name: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 can cover the display panel 1041, and when the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event, and then the processor 1080 provides corresponding visual output on the display panel 1041 according to the type of touch event. Although in the above embodiment, the touch panel 1031 and the display panel 1041 are realized as two independent components to realize the input and output functions of the mobile phone, in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone. Figure 3

[0182] ​The mobile phone can further include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor can include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 1041 according to the brightness of ambient light, and the proximity sensor can turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and when at rest, it can detect the magnitude and direction of gravity, which can be used for identifying the posture of the mobile phone (such as switching between landscape and portrait, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), and the like. As for other sensors that can also be configured on the mobile phone, such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, and the like, they will not be described here.

[0183] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can convert the received audio data into an electrical signal, transmit it to the speaker 1061, and convert it into a sound signal output by the speaker 1061; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then output to the processor 1080 for processing, and then transmitted to another mobile phone through the RF circuit 1010, or output to the memory 1020 for further processing.

[0184] WiFi belongs to a short-range wireless transmission technology, and the mobile phone can help the user to send and receive emails, browse web pages, and access streaming media through the WiFi module 1070, which provides the user with wireless broadband Internet access. Although Figure 3 The WiFi module 1070 is shown, but it can be understood that it does not belong to the essential components of the mobile phone, and can be omitted as needed without changing the essence of the application.

[0185] The processor 1080 is the control center of the mobile phone, which connects all parts of the mobile phone through various interfaces and lines, executes various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 1020 and calling data stored in the memory 1020, and thus monitors the mobile phone as a whole. Optionally, the processor 1080 can include one or more processing units; preferably, the processor 1080 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface, and the application program, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 1080.

[0186] The mobile phone further includes a power supply 1090 (such as a battery) for supplying power to various components, and preferably, the power supply is logically connected to the processor 1080 through a power management system, so that the power management system can realize functions such as charge management, discharge management, and power consumption management.

[0187] Although not shown, the mobile phone can further include a camera, a Bluetooth module, and the like, which will not be described here.

[0188] In the embodiments of the present application, the processor 1080 included in the terminal further has the following functions:

[0189] Step S1, obtaining source speaker speech and target speaker speech;

[0190] It should be noted that first, the source speaker speech of the source speaker and the target speaker speech of the target speaker are collected respectively.

[0191] In order to ensure that the quality of the voice sample meets the requirements such as intelligibility and no obvious background noise, the source speaker speech and the target speaker speech can be further preprocessed such as denoising and normalization to improve the accuracy of subsequent processing.

[0192] The present application is based on the conditional VAE framework as shown in Figure 4 , which includes a content encoder E and a generator G. Assuming that the speech x is composed of content information C and timbre S, the content information C describes the language content between different speakers, and the timbre S describes the timbre between different speakers, that is, C=E c (x), S=E s (x).

[0193] Step S2, inputting the source speaker speech into a pre-trained self-supervised speech representation learning model to obtain a feature vector x ssl containing source content information C.

[0194] It should be noted that the HuBERT model is a self-supervised speech representation learning model. The full name of HuBERT is "Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units", that is, self-supervised speech representation learning by masked prediction of hidden units.

[0195] HuBERT uses a Transformer-based encoder, which is a common component of many advanced speech processing models currently, and can process sequence data and capture long-distance dependencies.

[0196] During training, a certain proportion of speech frames are randomly selected and replaced with a special [MASK] token, and the model needs to predict these masked frames.

[0197] Thanks to self-supervised learning, HuBERT does not require a large amount of labeled data, making it very useful in data-scarce situations.

[0198] The feature representation learned by HuBERT has good generalization ability and can adapt to different speech processing tasks.

[0199] HuBERT provides an end-to-end framework that can learn useful feature representations directly from raw speech waveforms.

[0200] Through the HuBERT model, a 256-dimensional feature vector x containing source content information C is learned from the source speaker's speech ssl .

[0201] The feature vector x output by the Hubert model ssl may need further optimization or selection to retain the most useful information.

[0202] Step S3, the feature vector x ssl is decoupled as a conditional constraint source content information C into the flow model, and trained using the KL divergence criterion to obtain the target content information;

[0203] It should be noted that by using the reversible characteristics of the flow model, relying on x ssl feature vector constraint content information C decoupling.

[0204] The decoupling process needs to ensure the integrity and independence of the content information, which may involve hyperparameter adjustment and model fine-tuning.

[0205] The flow model transforms the content information to a Gaussian distribution that is where the mean and variance are calculated by the residual block from the x ssl feature vector, and the residual block network structure is shown in Figure 5 .

[0206] In order to realize the decoupling of x ssl feature constraint content information C, both of the above two Gaussian distributions are similar, and the KL divergence criterion is used for training, as follows:

[0207]

[0208] Make full use of the reversible characteristics of Flow, and the KL divergence criterion also acts on the inverse process, as follows:

[0209]

[0210] Step S4, encoding the source speaker voice and the target speaker voice through an encoder to obtain first timbre information of the source speaker and second timbre information of the target speaker respectively;

[0211] It should be noted that, in order to model the timbre information to reduce the timbre leakage problem, the timbre information Suppose it obeys the standard normal distribution, and also use the KL divergence criterion to constrain Approximate standard normal distribution:

[0212]

[0213] The simultaneous recovery loss function is:

[0214]

[0215] Step S5, replacing the first timbre information of the source speaker with the second timbre information of the target speaker, and using the generator to reconstruct the second timbre information and the target content information to generate the target voice.

[0216] Further, step S1 specifically includes:

[0217] The source speaker voice and the target speaker voice are obtained respectively, and the source speaker voice and the target speaker voice are preprocessed by denoising and normalization.

[0218] It should be noted that, in order to ensure that the quality of the voice sample meets the requirements, such as intelligibility, no obvious background noise, etc., the source speaker voice and the target speaker voice can be further preprocessed, such as denoising, normalization, etc., to improve the accuracy of subsequent processing.

[0219] Further, step S3 specifically includes:

[0220] The feature vector x ssl As a constraint condition for decoupling the source content information C, it is input into the flow model;

[0221] In the flow model, the source content information C is decoupled and trained by using the KL divergence criterion to obtain the target content information.

[0222] Further, step S3 further includes:

[0223] In the decoupling training process, the source content information C is transformed into a Gaussian distribution The mean and variance contained in the Gaussian distribution are calculated by a preset residual block network.

[0224] It should be noted that the Flow model transforms the content information to a Gaussian distribution That is Where the mean, variance are calculated by x ssl The feature vector is calculated by the residual block, and the network structure of the residual block is shown in Figure 5 .

[0225] Further, the decoupled training specifically includes:

[0226] In the forward process, the flow model converts the source content information C to ssl under the constraint of the feature vector x

[0227] Calculate the first KL divergence between and the target Gaussian distribution;

[0228] In the reverse process, the flow model restores -1 to the training content information through the inverse transformation f .

[0229] Calculate the second KL divergence between the training content information and the source content information C;

[0230] Combine the first KL divergence and the second KL divergence into the loss function of the flow model, and iteratively train the flow model until the loss function converges.

[0231] Further, step S4 specifically includes:

[0232] Calculate the first linear spectrum and the second linear spectrum of the source speaker voice and the target speaker voice respectively;

[0233] Take the first linear spectrum and the second linear spectrum as the input of the encoder, and output the first timbre information S = E s (x) of the source speaker and the second timbre information

[0234] It should be noted that the fast Fourier transform (FFT) is performed on the source speaker voice and the target speaker voice to convert the time domain signal to the frequency domain.

[0235] The linear spectrum is extracted from the FFT result. The linear spectrum is a logarithmic representation of the power spectrum, and the power of each frequency point is usually represented using a logarithm of 10.

[0236] Apply the Mel filter bank (Mel-filter bank) to filter the linear spectrum to simulate the perceptual sensitivity of the human ear to different frequencies.

[0237] The filtered signal is then transformed using a Discrete Cosine Transform, typically of type DCT-II, to reduce the redundancy of the data.

[0238] Features are extracted from the DCT result, such as Mel Frequency Cepstral Coefficients (MFCCs) or other speech features.

[0239] The extracted features are used as input to an encoder. In a self-supervised learning model such as HuBERT, these features will be further processed to learn a representation of the speech.

[0240] The embodiments of the present application also provide a computer readable storage medium for storing program codes, the program codes being used to execute any one of the speech conversion methods described in the foregoing embodiments.

[0241] In the embodiments of the present application, a speech conversion method, device, equipment and computer readable storage medium are provided, based on a conditional VAE framework, timbre information and content information of speech are extracted by an encoder respectively, and then a generator is used to synthesize speech, in order to decouple speech into content information and timbre information, a pre-trained self-supervised speech representation learning model is introduced to improve the content decoupling capability, a feature vector x containing source content information C output by the self-supervised speech representation learning model is input into the flow model, and the flow model is used to decouple the content information C from the timbre information. ssl As a conditional constraint for decoupling the content information C, the flow model is used to guide the model parameter continuation by using two KL divergence criteria, to ensure lossless of the content information, and the timbre information is modeled as a standard normal distribution, to reduce the timbre leakage problem, so as to improve the decoupling capability of the content information and solve the technical problem of timbre leakage.

[0242] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein.

[0243] The terms "first", "second", "third", "fourth" and the like in the description of this application and in the claims of the foregoing drawings, if any, are used for distinguishing similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of data so designated is interchangeable under appropriate circumstances so that the embodiments of the application described herein can be practiced in other than the illustrated or described order. Furthermore, the terms "comprise", "comprising", "include", "including", and the like are intended to cover non-exclusive inclusion, such that processes, methods, systems, products, or devices that comprise, include, or the like, a series of steps or units are not necessarily limited to those explicitly listed, but can include other steps or units not expressly listed or inherent to such processes, methods, products, or devices.

[0244] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, and c can be single or multiple.

[0245] In several embodiments provided in the application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division, and actual implementation can have another division manner. For example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0246] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0247] In addition, each functional unit in each embodiment of the application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0248] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (English full name: Read-Only Memory, English abbreviation: ROM), a random access memory (English full name: Random Access Memory, English abbreviation: RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0249] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A voice conversion method characterized by, The method comprises the following steps: Step S1, obtaining source speaker voice and target speaker voice; Step S2, input the source speaker voice into the pre-trained self-supervised voice representation learning model to obtain a feature vector x containing source content information C ssl ; Step S3, obtaining the feature vector x ssl The conditional constraint source content information C is decoupled into the flow model, and the KL divergence criterion is used for training to obtain the target content information. Step S4, encoding the source speaker voice and the target speaker voice through an encoder to obtain first timbre information of the source speaker and second timbre information of the target speaker, respectively; Step S5, replacing the first timbre information of the source speaker with the second timbre information of the target speaker, and reconstructing the second timbre information and the target content information through a generator to generate target voice.

2. The voice conversion method of claim 1, wherein, The step S1 specifically comprises: The source speaker voice and the target speaker voice are obtained respectively, and the source speaker voice and the target speaker voice are preprocessed by denoising and normalization.

3. The voice conversion method of claim 1, wherein, The step S2 specifically comprises: inputting the source speaker voice into a pre-trained self-supervised speech representation learning model to obtain a feature vector x of a target dimension containing source content information C ssl wherein the self-supervised speech representation learning model is specifically a HuBERT model.

4. The voice conversion method of claim 1, wherein, The step S3 specifically comprises: The feature vector x ss l as a condition for decoupling the source content information C into the stream model; In the flow model, the source content information C is decoupled and trained by using a KL divergence criterion to obtain target content information.

5. The voice conversion method of claim 4, wherein, The step S3 further comprises: In the decoupled training process, the source content information C is transformed to a Gaussian distribution The mean and variance contained in the middle are calculated by the preset residual block network of the feature vector.

6. The voice conversion method of claim 5, wherein, The decoupled training specifically comprises: In the forward process, the flow model transforms the source content information C in a feature vector x ss under the constraints of the flow model f to Computing a first KL divergence between the target Gaussian distribution; In the reverse process, the flow model is transformed by the inverse transform f -1 The training content information is recovered as recovered as training content information; calculating a second KL divergence between the training content information and the source content information C; combining the first KL divergence and the second KL divergence as a loss function of the flow model, and iteratively training the flow model until the loss function converges.

7. The voice conversion method of claim 1, wherein, The step S4 specifically comprises: calculating first linear spectrum and second linear spectrum of the source speaker voice and the target speaker voice, respectively; outputting, by an encoder, first timbre information S of a source speaker as an input of the encoder s (x) and second timbre information of a target speaker 8. A voice conversion apparatus characterized by comprising: The method comprises the following steps: An acquisition unit is configured to acquire source speaker voice and target speaker voice; The first processing unit is configured to input the source speaker voice input into a pre-trained self-supervised speech representation learning model to obtain a feature vector x containing source content information C ssl ; a second processing unit configured to obtain a feature vector x ssl The conditional constraint source content information C is decoupled into the flow model, and the KL divergence criterion is used for training to obtain the target content information. A third processing unit is configured to encode the source speaker voice and the target speaker voice through an encoder to obtain first timbre information of the source speaker and second timbre information of the target speaker, respectively; A voice conversion unit is configured to replace the first timbre information of the source speaker with the second timbre information of the target speaker, and reconstruct the second timbre information and the target content information through a generator to generate target voice.

9. A voice conversion apparatus characterized by comprising: The device comprises a processor and a memory: The memory is configured to store program code and transmit the program code to the processor; The processor is configured to execute the voice conversion method according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store program code, and the program code is configured to execute the voice conversion method. The computer readable storage medium is configured to store program code, and the program code is configured to execute the voice conversion method.

Images