Voice conversion method, device and equipment and computer readable storage medium

Abstract

The invention discloses a voice conversion method, device and equipment and a computer readable storage medium, based on a conditional VAE framework, tone information and content information of voice are extracted by using an encoder, then the voice is synthesized through a generator, and meanwhile, in order to decouple the voice into the content information and the tone information, the voice is converted into the content information and the tone information. A pre-trained self-supervised speech representation learning model is introduced to improve the content decoupling capability, a feature vector xssl which is output by the self-supervised speech representation learning model and contains source content information C is used as decoupling of conditional constraint content information C through a stream model, the forward and reverse characteristics of the stream model are utilized, two KL divergence criteria are used for guiding model parameter continuation, and the self-supervised speech representation learning model is obtained. According to the method and the device, the content information is ensured to be lossless, the tone information is modeled as standard normal distribution, the problem of tone leakage is reduced, the decoupling capability of the content information is improved, and the technical problem of tone leakage is solved.

Description

Technical Field

[0001] The present application relates to the field of speech conversion technology, and in particular to a speech conversion method, device, equipment and computer-readable storage medium. Background Art

[0002] Voice conversion technology, also known as voice changing technology, is a fascinating and promising technology in the field of speech processing. It involves converting one person's voice into another person's voice while keeping the original language content intact.

[0003] Speech conversion technology has shown its importance in many fields, such as film and television post-production, speech synthesis, privacy protection, language learning, audio editing, and assisting people with disabilities. It can improve user experience, enhance privacy security, and provide technical support for the creative industry.

[0004] Traditional speech conversion methods rely on statistical models and require the collection of parallel speech datasets of source and target speakers. These data are used to train a conversion module that learns the acoustic feature mapping relationship between the source and target speakers.

[0005] With the development of deep learning technology, the field of speech conversion has undergone significant technological innovation. Modern speech conversion models use deep neural networks to more effectively capture and convert the characteristics of speech signals.

[0006] Existing research has begun to explore end-to-end speech conversion models that can directly convert from source speech to target speech without explicitly modeling intermediate features.

[0007] A key direction of modern speech conversion technology is to decouple the content information and timbre information in speech. This allows the model to process the semantic and acoustic features of speech separately, thereby performing speech conversion more flexibly. The timbre information is usually represented by a voiceprint vector (such as i-vector or d-vector), while the content information can be represented by phonetic posterior-grams or other acoustic features.

[0008] How to improve the decoupling capability of content information while reducing the problem of sound leakage is an urgent problem that technicians in this field need to solve. Summary of the invention

[0009] The present application provides a speech conversion method, apparatus, device and computer-readable storage medium, which improve the decoupling capability of content information and solve the technical problem of timbre leakage.

[0010] In view of this, the first aspect of the present application provides a voice conversion method, the method comprising:

[0011] Step S1, obtaining the speech of a source speaker and the speech of a target speaker;

[0012] Step S2: Input the source speaker's speech into the pre-trained self-supervised speech representation learning model to obtain a feature vector x containing the source content information C ssl ;

[0013] Step S3: transform the feature vector x ssl The decoupled input of the conditional constraint source content information C is input into the flow model and trained using the KL divergence criterion to obtain the target content information;

[0014] Step S4, encoding the source speaker's speech and the target speaker's speech by an encoder to obtain first timbre information of the source speaker and second timbre information of the target speaker respectively;

[0015] Step S5: Replace the first timbre information of the source speaker with the second timbre information of the target speaker, and use a generator to reconstruct the second timbre information and the target content information to generate a target speech.

[0016] Optionally, the step S1 specifically includes:

[0017] The source speaker's speech and the target speaker's speech are obtained respectively, and the source speaker's speech and the target speaker's speech are pre-processed by denoising and normalizing.

[0018] Optionally, the step S2 specifically includes:

[0019] The source speaker’s speech is used as the input of the pre-trained self-supervised speech representation learning model to obtain the feature vector x of the target dimension containing the source content information C ssl , wherein the self-supervised speech representation learning model is specifically a HuBERT model.

[0020] Optionally, the step S3 specifically includes:

[0021] The feature vector x ssl The condition that constrains the decoupling of the source content information C is input into the flow model;

[0022] In the flow model, the source content information C is decoupled and trained using the KL divergence criterion to obtain the target content information.

[0023] Optionally, step S3 further includes:

[0024] In the decoupling training process, the source content information C is transformed into a Gaussian distribution The mean and variance contained in are calculated by the feature vector through a preset residual block network.

[0025] Optionally, the decoupling training is specifically:

[0026] In the forward process, the flow model transforms the source content information C into the feature vector x by transforming f ss Under the constraint of l, the transformation to

[0027] calculate The first KL divergence with the target Gaussian distribution;

[0028] In the reverse process, the flow model is transformed by the inverse -1 Will Restore to training content information;

[0029] Calculating a second KL divergence between the training content information and the source content information C;

[0030] The first KL divergence and the second KL divergence are combined into a loss function of the flow model, and the flow model is iteratively trained until the loss function converges.

[0031] Optionally, the step S4 specifically includes:

[0032] Calculating a first linear spectrum and a second linear spectrum of the source speaker's speech and the target speaker's speech respectively;

[0033] The first linear spectrum and the second linear spectrum are used as inputs of an encoder, and the encoder outputs first timbre information S=E of the source speaker respectively. s (x) and the second timbre information of the target speaker

[0034] A second aspect of the present application provides a speech conversion device, the device comprising:

[0035] An acquisition unit, used for acquiring the speech of a source speaker and the speech of a target speaker;

[0036] The first processing unit is used to input the source speaker's speech into the pre-trained self-supervised speech representation learning model to obtain a feature vector x containing the source content information C ssl ;

[0037] The second processing unit is used to transform the feature vector x ssl The decoupled input of the conditional constraint source content information C is input into the flow model and trained using the KL divergence criterion to obtain the target content information;

[0038] a third processing unit, configured to encode the source speaker's speech and the target speaker's speech through an encoder to obtain first timbre information of the source speaker and second timbre information of the target speaker respectively;

[0039] The speech conversion unit is used to replace the first timbre information of the source speaker with the second timbre information of the target speaker, and to reconstruct the second timbre information and the target content information by using a generator to generate a target speech.

[0040] Optionally, the acquiring unit is specifically used for:

[0041] The source speaker's speech and the target speaker's speech are obtained respectively, and the source speaker's speech and the target speaker's speech are pre-processed by denoising and normalizing.

[0042] Optionally, the first processing unit is specifically configured to:

[0043] The source speaker’s speech is used as the input of the pre-trained self-supervised speech representation learning model to obtain the feature vector x of the target dimension containing the source content information C ss l, wherein the self-supervised speech representation learning model is specifically a HuBERT model.

[0044] Optionally, the second processing unit is specifically configured to:

[0045] The feature vector x ssl The condition that constrains the decoupling of the source content information C is input into the flow model;

[0046] In the flow model, the source content information C is decoupled and trained using the KL divergence criterion to obtain the target content information.

[0047] Optionally, the second processing unit is further used for:

[0048] In the decoupling training process, the source content information C is transformed into a Gaussian distribution The mean and variance contained in are calculated by the feature vector through a preset residual block network.

[0049] Optionally, the decoupling training is specifically:

[0050] In the forward process, the flow model transforms the source content information C into the feature vector x by transforming f ss Under the constraint of l, the transformation to

[0051] calculate The first KL divergence with the target Gaussian distribution;

[0052] In the reverse process, the flow model is transformed by the inverse -1 Will Restore to training content information;

[0053] Calculating a second KL divergence between the training content information and the source content information C;

[0054] The first KL divergence and the second KL divergence are combined into a loss function of the flow model, and the flow model is iteratively trained until the loss function converges.

[0055] Optionally, the third processing unit is specifically configured to:

[0056] Calculating a first linear spectrum and a second linear spectrum of the source speaker's speech and the target speaker's speech respectively;

[0057] The first linear spectrum and the second linear spectrum are used as inputs of an encoder, and the encoder outputs first timbre information S=E of the source speaker respectively. s (x) and the second timbre information of the target speaker

[0058] A third aspect of the present application provides a speech conversion device, the device comprising a processor and a memory:

[0059] The memory is used to store program code and transmit the program code to the processor;

[0060] The processor is used to execute the steps of the method for voice conversion as described in the first aspect according to the instructions in the program code.

[0061] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in the first aspect above.

[0062] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0063] In the present application, a speech conversion method, apparatus, device and computer-readable storage medium are provided. Based on the conditional VAE framework, an encoder is used to extract the timbre information and content information of the speech respectively, and then the speech is synthesized by a generator. At the same time, in order to decouple the speech into content information and timbre information, a pre-trained self-supervised speech representation learning model is introduced to improve the content decoupling capability. The feature vector x containing the source content information C output by the self-supervised speech representation learning model is converted into a feature vector x containing the source content information C by a stream model. sslAs a decoupling of the conditionally constrained content information C, the forward and reverse characteristics of the flow model are utilized, and two KL divergence criteria are used to guide the connection of model parameters to ensure the losslessness of the content information. At the same time, the timbre information is modeled as a standard normal distribution to reduce the problem of timbre leakage, thereby improving the decoupling ability of the content information and solving the technical problem of timbre leakage. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A method flow chart of the voice conversion method in an embodiment of the present application;

[0065] Figure 2 This is a schematic diagram of the structure of the speech conversion device in the embodiment of the present application;

[0066] Figure 3 This is a schematic diagram of the structure of a voice conversion device in an embodiment of the present application;

[0067] Figure 4 This is a diagram of the conditional VAE framework structure in the embodiment of this application;

[0068] Figure 5 This is a diagram of the residual block network structure in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0070] The present application designs a speech conversion method, apparatus, device and computer-readable storage medium. Based on the conditional VAE framework, an encoder is used to extract the timbre information and content information of the speech respectively, and then the speech is synthesized through a generator. At the same time, in order to decouple the speech into content information and timbre information, a pre-trained Huber model is introduced to improve the content decoupling capability. The feature vector x containing the source content information C output by the Huber model is converted into the feature vector x by the flow model. ssl As a decoupling of the conditional constraint content information C, the forward and reverse characteristics of the flow model are utilized, and two KL divergence criteria are used to guide the connection of model parameters to ensure the losslessness of content information. At the same time, the timbre information is modeled as a standard normal distribution to reduce the problem of timbre leakage, thereby improving the decoupling ability of content information and solving the technical problem of timbre leakage.

[0071] For easier understanding, see Figure 1 , Figure 1is a flow chart of the method for voice conversion in the embodiment of the present application, such as Figure 1 As shown, specifically:

[0072] Step S1, obtaining the speech of a source speaker and the speech of a target speaker;

[0073] It should be noted that first, the source speaker's speech and the target speaker's speech are collected respectively.

[0074] In order to ensure that the quality of the speech sample meets the requirements, such as clarity and absence of obvious background noise, the source speaker's speech and the target speaker's speech can be further preprocessed, such as denoising and normalization, to improve the accuracy of subsequent processing.

[0075] This application is based on Figure 4 The conditional VAE framework shown in the figure contains a content encoder E and a generator G. Assume that speech x consists of content information C and timbre S. The content information C describes the language content between different speakers, and the timbre S describes the timbre between different speakers, that is, C = E c (x),S=E s (x).

[0076] Step S2: Input the source speaker's speech into the pre-trained self-supervised speech representation learning model to obtain a feature vector x containing the source content information C ssl ;

[0077] It should be noted that the HuBERT model is a self-supervised speech representation learning model. The full name of HuBERT is "Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units", which means self-supervised speech representation learning through mask prediction of hidden units.

[0078] HuBERT uses a Transformer-based encoder, a common component of many current advanced speech processing models that can process sequence data and capture long-range dependencies.

[0079] During training, a certain proportion of speech frames are randomly selected and replaced with a special [MASK] tag, and the model needs to predict these masked frames.

[0080] Due to its self-supervised learning, HuBERT does not require a large amount of labeled data, which makes it very useful even in data-scarce situations.

[0081] The feature representation learned by HuBERT has good generalization ability and can adapt to different speech processing tasks.

[0082] HuBERT provides an end-to-end framework that can learn useful feature representations directly from raw speech waveforms.

[0083] The HuBERT model is used to learn a 256-dimensional feature vector x containing the source content information C from the source speaker’s speech. ssl .

[0084] The feature vector x output by the Hubert model ssl Further optimization or selection may be required to retain the most useful information.

[0085] Step S3: transform the feature vector x ssl The decoupled input of the conditional constraint source content information C is input into the flow model and trained using the KL divergence criterion to obtain the target content information;

[0086] It should be noted that by using the reversible characteristics of the flow model, relying on x ssl Decoupling of the feature vector constraint content information C.

[0087] The decoupling process needs to ensure the integrity and independence of content information, which may involve hyperparameter adjustment and model fine-tuning.

[0088] The Flow model converts content information Transform to Gaussian distribution Right now The mean and variance By x ssl The feature vector is calculated by the residual block, and the residual block network structure is as follows Figure 5 shown.

[0089] In order to achieve x ssl The decoupling of feature constraint content information C satisfies the similarity of the above two Gaussian distributions and uses the KL divergence criterion for training, as shown in the following formula:

[0090]

[0091] Taking full advantage of the reversible nature of Flow, the KL divergence criterion also works on the inverse process, as shown in the following formula:

[0092]

[0093] Step S4, encoding the source speaker's speech and the target speaker's speech by an encoder to obtain first timbre information of the source speaker and second timbre information of the target speaker respectively;

[0094] It should be noted that in order to model the timbre information and reduce the timbre leakage problem, the timbre information Assuming that it obeys the standard normal distribution, the KL divergence criterion is also used to constrain Approximate standard normal distribution:

[0095]

[0096] At the same time, the recovery loss function is:

[0097]

[0098] Step S5: Replace the first timbre information of the source speaker with the second timbre information of the target speaker, and use a generator to reconstruct the second timbre information and the target content information to generate the target speech.

[0099] Furthermore, step S1 specifically includes:

[0100] The source speaker's speech and the target speaker's speech are obtained respectively, and the source speaker's speech and the target speaker's speech are preprocessed by denoising and normalizing.

[0101] It should be noted that in order to ensure that the quality of the speech sample meets the requirements, such as clarity and absence of obvious background noise, the source speaker's speech and the target speaker's speech can be further preprocessed, such as denoising and normalization, to improve the accuracy of subsequent processing.

[0102] Furthermore, step S3 specifically includes:

[0103] The feature vector x ssl The condition that constrains the decoupling of the source content information C is input into the flow model;

[0104] In the flow model, the KL divergence criterion is used to decouple the source content information C for training to obtain the target content information.

[0105] Furthermore, step S3 also includes:

[0106] During the decoupled training process, the source content information C is transformed into a Gaussian distribution The mean and variance contained in are the feature vectors calculated by the preset residual block network.

[0107] It should be noted that the Flow model converts content information Transform to Gaussian distribution Right now The mean and variance By x ssl The feature vector is calculated by the residual block, and the residual block network structure is as follows Figure 5 shown.

[0108] Furthermore, decoupling training is specifically as follows:

[0109] In the forward process, the flow model transforms the source content information C into the feature vector x ssl Under the constraint of

[0110] calculate The first KL divergence with the target Gaussian distribution;

[0111] In the reverse process, the flow model is transformed by the inverse transformation f -1 Will Restore to training content information;

[0112] Calculate the second KL divergence between the training content information and the source content information C;

[0113] The first KL divergence and the second KL divergence are combined as the loss function of the flow model, and the flow model is iteratively trained until the loss function converges.

[0114] Furthermore, step S4 specifically includes:

[0115] Calculating a first linear spectrum and a second linear spectrum of the source speaker's speech and the target speaker's speech respectively;

[0116] The first linear spectrum and the second linear spectrum are used as the input of the encoder, and the encoder outputs the first timbre information S=E of the source speaker respectively. s (x) and the second timbre information of the target speaker

[0117] It should be noted that a Fast Fourier Transform (FFT) is performed on the source speaker's speech and the target speaker's speech to convert the time domain signal into the frequency domain.

[0118] Extract the linear spectrum from the FFT result. The linear spectrum is the logarithmic representation of the power spectrum, usually using 10 times the logarithm to represent the power at each frequency point.

[0119] A Mel-filterbank is applied to filter the linear spectrum to simulate the human ear's perception sensitivity to different frequencies.

[0120] The filtered signal is subjected to a discrete cosine transform, usually DCT-II type, to reduce data redundancy.

[0121] Extract features such as Mel-frequency cepstral coefficients (MFCCs) or other speech features from the DCT results.

[0122] The extracted features are used as input to the encoder. In self-supervised learning models such as HuBERT, these features will be further processed to learn representations of speech.

[0123] See also Figure 2 , Figure 2 Schematic diagram of the structure of the speech conversion device in the embodiment of the present application. Figure 2 As shown, specifically:

[0124] An acquisition unit 201 is used to acquire the speech of a source speaker and the speech of a target speaker;

[0125] It should be noted that first, the source speaker's speech and the target speaker's speech are collected respectively.

[0126] In order to ensure that the quality of the speech sample meets the requirements, such as clarity and absence of obvious background noise, the source speaker's speech and the target speaker's speech can be further preprocessed, such as denoising and normalization, to improve the accuracy of subsequent processing.

[0127] This application is based on Figure 4 The conditional VAE framework shown in the figure contains a content encoder E and a generator G. Assume that speech x consists of content information C and timbre S. The content information C describes the language content between different speakers, and the timbre S describes the timbre between different speakers, that is, C = E c (x),S=E s (x).

[0128] The first processing unit 202 is used to input the source speaker's speech into the pre-trained self-supervised speech representation learning model to obtain a feature vector x containing the source content information C ssl ;

[0129] It should be noted that the HuBERT model is a self-supervised speech representation learning model. The full name of HuBERT is "Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units", which means self-supervised speech representation learning through mask prediction of hidden units.

[0130] HuBERT uses a Transformer-based encoder, a common component of many current advanced speech processing models that can process sequence data and capture long-range dependencies.

[0131] During training, a certain proportion of speech frames are randomly selected and replaced with a special [MASK] tag, and the model needs to predict these masked frames.

[0132] Due to its self-supervised learning, HuBERT does not require a large amount of labeled data, which makes it very useful even in data-scarce situations.

[0133] The feature representation learned by HuBERT has good generalization ability and can adapt to different speech processing tasks.

[0134] HuBERT provides an end-to-end framework that can learn useful feature representations directly from raw speech waveforms.

[0135] The HuBERT model is used to learn a 256-dimensional feature vector x containing the source content information C from the source speaker’s speech. ssl .

[0136] The feature vector x output by the Hubert model ssl Further optimization or selection may be required to retain the most useful information.

[0137] The second processing unit 203 is used to convert the feature vector x ssl The decoupled input of the conditional constraint source content information C is input into the flow model and trained using the KL divergence criterion to obtain the target content information;

[0138] It should be noted that by using the reversible characteristics of the flow model, relying on x ssl Decoupling of the feature vector constraint content information C.

[0139] The decoupling process needs to ensure the integrity and independence of content information, which may involve hyperparameter adjustment and model fine-tuning.

[0140] The Flow model converts content information Transform to Gaussian distribution Right now The mean and variance By x ssl The feature vector is calculated by the residual block, and the residual block network structure is as follows Figure 5 shown.

[0141] In order to achieve x ssl The decoupling of feature constraint content information C satisfies the similarity of the above two Gaussian distributions and uses the KL divergence criterion for training, as shown in the following formula:

[0142]

[0143] Taking full advantage of the reversible nature of Flow, the KL divergence criterion also works on the inverse process, as shown in the following formula:

[0144]

[0145] The third processing unit 204 is used to encode the source speaker's speech and the target speaker's speech through an encoder to obtain first timbre information of the source speaker and second timbre information of the target speaker respectively;

[0146] It should be noted that in order to model the timbre information and reduce the timbre leakage problem, the timbre information Assuming that it obeys the standard normal distribution, the KL divergence criterion is also used to constrain Approximate standard normal distribution:

[0147]

[0148] At the same time, the recovery loss function is:

[0149]

[0150] The speech conversion unit 205 is used to replace the first timbre information of the source speaker with the second timbre information of the target speaker, and reconstruct the second timbre information and the target content information by using a generator to generate the target speech.

[0151] Furthermore, the acquisition unit 201 is specifically used for:

[0152] The source speaker's speech and the target speaker's speech are obtained respectively, and the source speaker's speech and the target speaker's speech are preprocessed by denoising and normalizing.

[0153] It should be noted that in order to ensure that the quality of the speech sample meets the requirements, such as clarity and absence of obvious background noise, the source speaker's speech and the target speaker's speech can be further preprocessed, such as denoising and normalization, to improve the accuracy of subsequent processing.

[0154] Further, the second processing unit 203 is specifically configured to:

[0155] The feature vector x ssl The condition that constrains the decoupling of the source content information C is input into the flow model;

[0156] In the flow model, the KL divergence criterion is used to decouple the source content information C for training to obtain the target content information.

[0157] Furthermore, the second processing unit 203 is further configured to:

[0158] During the decoupled training process, the source content information C is transformed into a Gaussian distribution The mean and variance contained in are the feature vectors calculated by the preset residual block network.

[0159] It should be noted that the Flow model converts content information Transform to Gaussian distribution Right now The mean and variance By x sslThe feature vector is calculated by the residual block, and the residual block network structure is as follows Figure 5 shown.

[0160] Furthermore, decoupling training is specifically as follows:

[0161] In the forward process, the flow model transforms the source content information C into the feature vector x ssl Under the constraint of

[0162] calculate The first KL divergence with the target Gaussian distribution;

[0163] In the reverse process, the flow model is transformed by the inverse -1 Will Restore to training content information;

[0164] Calculate the second KL divergence between the training content information and the source content information C;

[0165] The first KL divergence and the second KL divergence are combined as the loss function of the flow model, and the flow model is iteratively trained until the loss function converges.

[0166] Further, the third processing unit 204 is specifically configured to:

[0167] Calculating a first linear spectrum and a second linear spectrum of the source speaker's speech and the target speaker's speech respectively;

[0168] The first linear spectrum and the second linear spectrum are used as the input of the encoder, and the encoder outputs the first timbre information S=E of the source speaker respectively. s (x) and the second timbre information of the target speaker

[0169] It should be noted that a Fast Fourier Transform (FFT) is performed on the source speaker's speech and the target speaker's speech to convert the time domain signal into the frequency domain.

[0170] Extract the linear spectrum from the FFT result. The linear spectrum is the logarithmic representation of the power spectrum, usually using 10 times the logarithm to represent the power at each frequency point.

[0171] A Mel-filterbank is applied to filter the linear spectrum to simulate the human ear's perception sensitivity to different frequencies.

[0172] The filtered signal is subjected to a discrete cosine transform, usually DCT-II type, to reduce data redundancy.

[0173] Extract features such as Mel-frequency cepstral coefficients (MFCCs) or other speech features from the DCT results.

[0174] The extracted features are used as input to the encoder. In self-supervised learning models such as HuBERT, these features will be further processed to learn representations of speech.

[0175] The present application also provides another voice conversion device, such as Figure 3 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), a sales terminal (English full name: Point of Sales, English abbreviation: POS), a car computer, etc., taking the terminal as a mobile phone as an example:

[0176] Figure 3 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal provided in an embodiment of the present application. Figure 3 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art can understand that Figure 3 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0177] Combine the following Figure 3 A detailed introduction to the various components of the mobile phone:

[0178] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (full name: Low Noise Amplifier, English abbreviation: LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communications can use any communication standard or protocol, including but not limited to Global System of Mobile communications (Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, English abbreviation: LTE), e-mail, Short Messaging Service (SMS), etc.

[0179] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0180] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0181] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (full name in English: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (full name in English: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 3 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0182] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0183] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.

[0184] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access. Figure 3 A WiFi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0185] The processor 1080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 1020 and calling data stored in the memory 1020, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0186] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0187] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0188] In the embodiment of the present application, the processor 1080 included in the terminal also has the following functions:

[0189] Step S1, obtaining the speech of a source speaker and the speech of a target speaker;

[0190] It should be noted that first, the source speaker's speech and the target speaker's speech are collected respectively.

[0191] In order to ensure that the quality of the speech sample meets the requirements, such as clarity and absence of obvious background noise, the source speaker's speech and the target speaker's speech can be further preprocessed, such as denoising and normalization, to improve the accuracy of subsequent processing.

[0192] This application is based on Figure 4 The conditional VAE framework shown in the figure contains a content encoder E and a generator G. Assume that speech x consists of content information C and timbre S. The content information C describes the language content between different speakers, and the timbre S describes the timbre between different speakers, that is, C = E c (x),S=E s (x).

[0193] Step S2: Input the source speaker's speech into the pre-trained self-supervised speech representation learning model to obtain a feature vector x containing the source content information C ssl ;

[0194] It should be noted that the HuBERT model is a self-supervised speech representation learning model. The full name of HuBERT is "Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units", which means self-supervised speech representation learning through mask prediction of hidden units.

[0195] HuBERT uses a Transformer-based encoder, a common component of many current advanced speech processing models that can process sequence data and capture long-range dependencies.

[0196] During training, a certain proportion of speech frames are randomly selected and replaced with a special [MASK] tag, and the model needs to predict these masked frames.

[0197] Due to its self-supervised learning, HuBERT does not require a large amount of labeled data, which makes it very useful even in data-scarce situations.

[0198] The feature representation learned by HuBERT has good generalization ability and can adapt to different speech processing tasks.

[0199] HuBERT provides an end-to-end framework that can learn useful feature representations directly from raw speech waveforms.

[0200] The HuBERT model is used to learn a 256-dimensional feature vector x containing the source content information C from the source speaker’s speech. ssl .

[0201] The feature vector x output by the Hubert model ssl Further optimization or selection may be required to retain the most useful information.

[0202] Step S3: transform the feature vector x ssl The decoupled input of the conditional constraint source content information C is input into the flow model and trained using the KL divergence criterion to obtain the target content information;

[0203] It should be noted that by using the reversible characteristics of the flow model, relying on x ssl Decoupling of the feature vector constraint content information C.

[0204] The decoupling process needs to ensure the integrity and independence of content information, which may involve hyperparameter adjustment and model fine-tuning.

[0205] The Flow model converts content information Transform to Gaussian distribution Right now The mean and variance By x ssl The feature vector is calculated by the residual block, and the residual block network structure is as follows Figure 5 shown.

[0206] In order to achieve x ssl The decoupling of feature constraint content information C satisfies the similarity of the above two Gaussian distributions and uses the KL divergence criterion for training, as shown in the following formula:

[0207]

[0208] Taking full advantage of the reversible nature of Flow, the KL divergence criterion also works on the inverse process, as shown in the following formula:

[0209]

[0210] Step S4, encoding the source speaker's speech and the target speaker's speech by an encoder to obtain first timbre information of the source speaker and second timbre information of the target speaker respectively;

[0211] It should be noted that in order to model the timbre information and reduce the timbre leakage problem, the timbre information Assuming that it obeys the standard normal distribution, the KL divergence criterion is also used to constrain Approximate standard normal distribution:

[0212]

[0213] At the same time, the recovery loss function is:

[0214]

[0215] Step S5: Replace the first timbre information of the source speaker with the second timbre information of the target speaker, and use a generator to reconstruct the second timbre information and the target content information to generate the target speech.

[0216] Furthermore, step S1 specifically includes:

[0217] The source speaker's speech and the target speaker's speech are obtained respectively, and the source speaker's speech and the target speaker's speech are preprocessed by denoising and normalizing.

[0218] It should be noted that in order to ensure that the quality of the speech sample meets the requirements, such as clarity and absence of obvious background noise, the source speaker's speech and the target speaker's speech can be further preprocessed, such as denoising and normalization, to improve the accuracy of subsequent processing.

[0219] Furthermore, step S3 specifically includes:

[0220] The feature vector x ssl The condition that constrains the decoupling of the source content information C is input into the flow model;

[0221] In the flow model, the KL divergence criterion is used to decouple the source content information C for training to obtain the target content information.

[0222] Furthermore, step S3 also includes:

[0223] During the decoupled training process, the source content information C is transformed into a Gaussian distribution The mean and variance contained in are the feature vectors calculated by the preset residual block network.

[0224] It should be noted that the Flow model converts content information Transform to Gaussian distribution Right now The mean and variance By x ssl The feature vector is calculated by the residual block, and the residual block network structure is as follows Figure 5 shown.

[0225] Furthermore, decoupling training is specifically as follows:

[0226] In the forward process, the flow model transforms the source content information C into the feature vector x ssl Under the constraint of

[0227] calculate The first KL divergence with the target Gaussian distribution;

[0228] In the reverse process, the flow model is transformed by the inverse -1 Will Restore to training content information;

[0229] Calculate the second KL divergence between the training content information and the source content information C;

[0230] The first KL divergence and the second KL divergence are combined as the loss function of the flow model, and the flow model is iteratively trained until the loss function converges.

[0231] Furthermore, step S4 specifically includes:

[0232] Calculating a first linear spectrum and a second linear spectrum of the source speaker's speech and the target speaker's speech respectively;

[0233] The first linear spectrum and the second linear spectrum are used as the input of the encoder, and the encoder outputs the first timbre information S=E of the source speaker respectively. s (x) and the second timbre information of the target speaker

[0234] It should be noted that a Fast Fourier Transform (FFT) is performed on the source speaker's speech and the target speaker's speech to convert the time domain signal into the frequency domain.

[0235] Extract the linear spectrum from the FFT result. The linear spectrum is the logarithmic representation of the power spectrum, usually using 10 times the logarithm to represent the power at each frequency point.

[0236] A Mel-filterbank is applied to filter the linear spectrum to simulate the human ear's perception sensitivity to different frequencies.

[0237] The filtered signal is subjected to a discrete cosine transform, usually DCT-II type, to reduce data redundancy.

[0238] Extract features such as Mel-frequency cepstral coefficients (MFCCs) or other speech features from the DCT results.

[0239] The extracted features are used as input to the encoder. In self-supervised learning models such as HuBERT, these features will be further processed to learn representations of speech.

[0240] The embodiment of the present application also provides a computer-readable storage medium for storing program code, wherein the program code is used to execute any implementation of a speech conversion method described in the aforementioned embodiments.

[0241] In the embodiment of the present application, a speech conversion method, apparatus, device and computer-readable storage medium are provided. Based on the conditional VAE framework, an encoder is used to extract the timbre information and content information of the speech respectively, and then the speech is synthesized by a generator. At the same time, in order to decouple the speech into content information and timbre information, a pre-trained self-supervised speech representation learning model is introduced to improve the content decoupling capability. The feature vector x containing the source content information C output by the self-supervised speech representation learning model is converted into the feature vector x of the source content information C through the stream model. ssl As a decoupling of the conditionally constrained content information C, the forward and reverse characteristics of the flow model are utilized, and two KL divergence criteria are used to guide the connection of model parameters to ensure the losslessness of the content information. At the same time, the timbre information is modeled as a standard normal distribution to reduce the problem of timbre leakage, thereby improving the decoupling ability of the content information and solving the technical problem of timbre leakage.

[0242] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0243] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0244] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0245] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0246] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0247] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0248] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0249] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for voice conversion, It is characterized in that include: Step S1, obtaining the speech of a source speaker and the speech of a target speaker; Step S2: Input the source speaker's speech into the pre-trained self-supervised speech representation learning model to obtain a feature vector x containing the source content information C ssl ; Step S3: transform the feature vector x ssl The decoupled input of the conditional constraint source content information C is input into the flow model and trained using the KL divergence criterion to obtain the target content information; Step S4, encoding the source speaker's speech and the target speaker's speech by an encoder to obtain first timbre information of the source speaker and second timbre information of the target speaker respectively; Step S5: Replace the first timbre information of the source speaker with the second timbre information of the target speaker, and use a generator to reconstruct the second timbre information and the target content information to generate a target speech.

2. The voice conversion method according to claim 1, It is characterized in that The step S1 specifically includes: The source speaker's speech and the target speaker's speech are obtained respectively, and the source speaker's speech and the target speaker's speech are pre-processed by denoising and normalizing.

3. The voice conversion method according to claim 1, It is characterized in that The step S2 specifically includes: The source speaker’s speech is used as the input of the pre-trained self-supervised speech representation learning model to obtain the feature vector x of the target dimension containing the source content information C ssl , wherein the self-supervised speech representation learning model is specifically a HuBERT model.

4. The voice conversion method according to claim 1, It is characterized in that The step S3 specifically includes: The feature vector x ss l is input into the flow model as a condition for decoupling the source content information C; In the flow model, the source content information C is decoupled and trained using the KL divergence criterion to obtain the target content information.

5. The voice conversion method according to claim 4, It is characterized in that The step S3 further comprises: In the decoupling training process, the source content information C is transformed into a Gaussian distribution The mean and variance contained in are calculated by the feature vector through a preset residual block network.

6. The voice conversion method according to claim 5, It is characterized in that The decoupling training is specifically as follows: In the forward process, the flow model transforms the source content information C into the feature vector x by transforming f ss Under the constraint of l, the transformation to calculate The first KL divergence with the target Gaussian distribution; In the reverse process, the flow model is transformed by the inverse -1 Will Restore to training content information; Calculating a second KL divergence between the training content information and the source content information C; The first KL divergence and the second KL divergence are combined into a loss function of the flow model, and the flow model is iteratively trained until the loss function converges.

7. The voice conversion method according to claim 1, It is characterized in that The step S4 specifically includes: Calculating a first linear spectrum and a second linear spectrum of the source speaker's speech and the target speaker's speech respectively; The first linear spectrum and the second linear spectrum are used as inputs of an encoder, and the encoder outputs first timbre information S=E of the source speaker respectively. s (x) and the second timbre information of the target speaker 8. A voice conversion device, It is characterized in that include: An acquisition unit, used for acquiring the speech of a source speaker and the speech of a target speaker; The first processing unit is used to input the source speaker's speech into the pre-trained self-supervised speech representation learning model to obtain a feature vector x containing the source content information C ssl ; The second processing unit is used to transform the feature vector x ssl The decoupled input of the conditional constraint source content information C is input into the flow model and trained using the KL divergence criterion to obtain the target content information; a third processing unit, configured to encode the source speaker's speech and the target speaker's speech through an encoder to obtain first timbre information of the source speaker and second timbre information of the target speaker respectively; The speech conversion unit is used to replace the first timbre information of the source speaker with the second timbre information of the target speaker, and to reconstruct the second timbre information and the target content information by using a generator to generate a target speech.

9. A speech conversion device, It is characterized in that The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the speech conversion method according to any one of claims 1 to 7 according to the instructions in the program code.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium is used to store program codes, and the program codes are used to execute the speech conversion method according to any one of claims 1 to 7.

Images