Voice changing method and device

Abstract

The present invention discloses a voice changing method and device, the method comprising: receiving a source speaker's sentence; extracting speech recognition acoustic features and speech synthesis acoustic features from the source speaker's sentence; obtaining speech recognition hidden features using the speech recognition acoustic features; obtaining speech synthesis coding features using the speech synthesis acoustic features; inputting the speech recognition hidden features and the speech synthesis coding features into a pre-constructed timbre conversion model corresponding to a specific target speaker to obtain the speech synthesis acoustic features of the specific target speaker; and generating a specific target speaker audio signal using the speech synthesis acoustic features of the specific target speaker. The scheme of the present invention can realize the transformation of any source speaker's voice to the target speaker's voice, and has a good voice changing effect.

Description

Technical Field

[0001] The present invention relates to the field of speech signal processing, and in particular to a voice changing method and device. Background Art

[0002] At present, with the development of speech synthesis technology, how to make synthesized speech natural, diversified and personalized has become a hot topic in current speech technology research, and voice changing technology is one of the ways to make synthesized speech diversified and personalized. Voice changing technology mainly refers to the technology that retains the semantic content of speech signals but changes the characteristics of the speaker's voice, making someone's voice sound like another person's voice. From the perspective of speaker conversion, voice changing technology is usually divided into two ways: one is voice conversion between non-specific people, such as conversion between male and female voices, conversion between different age groups, etc.; the other is voice conversion between specific people, such as converting the voice of speaker A to the voice of speaker B.

[0003] The traditional voice changing method to achieve the timbre conversion of any source speaker to the target speaker is usually based on speech recognition technology, using DTW (Dynamic Time Warping) or attention mechanism to align the parallel corpus, and then perform timbre conversion. This processing method requires collecting parallel corpus of the source speaker and the target speaker when training the conversion model, that is, audio corpus with the same content, and using the aligned spectral features to train the conversion model; when performing audio conversion, the spectral features extracted from the audio data of the source speaker are converted through the conversion model, and the fundamental frequency features are linearly stretched, and the non-periodic components are not changed. The voice changing effect obtained by this voice changing processing method is not good, especially the converted sound cannot well reflect the rhythm, emotion and other characteristics of the source speaker. Summary of the invention

[0004] The embodiment of the present invention provides a voice changing method and device to improve the voice changing effect and make the rhythm and emotion of the converted voice closer to the voice characteristics of the original speaker.

[0005] To this end, the present invention provides the following technical solutions:

[0006] A voice changing method, the method comprising:

[0007] Receiving a source speaker sentence;

[0008] Extracting speech recognition acoustic features and speech synthesis acoustic features from the source speaker's sentence;

[0009] Obtaining speech recognition latent features using the speech recognition acoustic features;

[0010] Obtaining speech synthesis coding features using the speech synthesis acoustic features;

[0011] Inputting the speech recognition latent features and the speech synthesis coding features into a pre-built timbre conversion model corresponding to a specific target speaker to obtain the speech synthesis acoustic features of the specific target speaker;

[0012] An audio signal of the specific target speaker is generated by using the speech synthesis acoustic features of the specific target speaker.

[0013] Optionally, the method further comprises constructing the timbre conversion model corresponding to the specific target speaker in the following manner:

[0014] Collect audio data from a specific target speaker;

[0015] The audio data of the specific target speaker is used to adaptively train a universal voice change model pre-constructed based on the audio data of multiple speakers to obtain a timbre conversion model corresponding to the specific target speaker.

[0016] Optionally, the method further includes: constructing a universal voice changing model based on audio data of multiple speakers, specifically including:

[0017] Collect audio data from multiple speakers as training data;

[0018] Extracting speech recognition acoustic features and speech synthesis acoustic features from the training data;

[0019] Obtaining speech recognition latent features using the speech recognition acoustic features;

[0020] Obtaining speech synthesis coding features using the speech synthesis acoustic features;

[0021] The universal voice change model is trained by utilizing the speech recognition hidden layer features, speech synthesis coding features and the speech synthesis acoustic features.

[0022] Optionally, the obtaining the speech recognition hidden features by using the speech recognition acoustic features includes:

[0023] The speech recognition acoustic features are input into a speech recognition model to obtain speech recognition latent features.

[0024] Optionally, obtaining the speech synthesis coding feature by using the speech synthesis acoustic feature includes:

[0025] The speech synthesis acoustic features are input into a pre-built reference coding model to obtain speech synthesis coding features.

[0026] Optionally, the reference coding model is a neural network model.

[0027] Optionally, the speech recognition acoustic features include any one or more of the following: Mel-frequency cepstral coefficients, perceptual linear prediction parameters.

[0028] Optionally, the speech synthesis acoustic features include: Mel spectrum.

[0029] A voice changing device, comprising:

[0030] A receiving module, used for receiving a source speaker's sentence;

[0031] A feature extraction module, used to extract speech recognition acoustic features and speech synthesis acoustic features from the source speaker's sentence;

[0032] A hidden layer feature acquisition module, used to obtain speech recognition hidden layer features using the speech recognition acoustic features;

[0033] A coding feature acquisition module, used to obtain speech synthesis coding features using the speech synthesis acoustic features;

[0034] A feature conversion module, used for inputting the speech recognition hidden layer features and the speech synthesis coding features into a pre-built timbre conversion model corresponding to a specific target speaker, to obtain the speech synthesis acoustic features of the specific target speaker;

[0035] The speech synthesis module is used to generate an audio signal of the specific target speaker by using the speech synthesis acoustic features of the specific target speaker.

[0036] Optionally, the device further comprises: a timbre conversion model construction module, used to construct a timbre conversion model corresponding to a specific target speaker;

[0037] The timbre conversion model building module comprises:

[0038] A target data collection unit, used to collect audio data of a specific target speaker;

[0039] The model training unit is used to adaptively train a universal voice changing model pre-built based on the audio data of multiple speakers using the audio data of the specific target speaker, so as to obtain a timbre conversion model corresponding to the specific target speaker.

[0040] Optionally, the device further comprises: a general model construction module, for constructing a general voice change model based on audio data of multiple speakers;

[0041] The general model building module includes:

[0042] A universal data collection unit, used for collecting audio data of multiple speakers as training data;

[0043] A feature extraction unit, used to extract speech recognition acoustic features and speech synthesis acoustic features from the training data;

[0044] A hidden layer feature acquisition unit, used to obtain a speech recognition hidden layer feature using the speech recognition acoustic feature;

[0045] A coding feature acquisition unit, used to obtain a speech synthesis coding feature using the speech synthesis acoustic feature;

[0046] The universal parameter training unit is used to train a universal voice changing model by utilizing the speech recognition hidden layer features, the speech synthesis coding features and the speech synthesis acoustic features.

[0047] Optionally, the hidden layer feature acquisition unit is specifically used to input the speech recognition acoustic feature into a speech recognition model to obtain the speech recognition hidden layer feature.

[0048] Optionally, the coding feature acquisition unit is specifically used to input the speech synthesis acoustic feature into a pre-built reference coding model to obtain the speech synthesis coding feature.

[0049] Optionally, the reference coding model is a neural network model.

[0050] Optionally, the speech recognition acoustic features include any one or more of the following: Mel-frequency cepstral coefficients, perceptual linear prediction parameters.

[0051] Optionally, the speech synthesis acoustic features include: Mel spectrum.

[0052] An electronic device, comprising: one or more processors and a memory;

[0053] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the aforementioned method.

[0054] A readable storage medium stores instructions thereon, wherein the instructions are executed to implement the above-mentioned method.

[0055] The voice changing method and device provided by the embodiment of the present invention pre-builds a timbre conversion model corresponding to a specific target speaker, extracts speech recognition acoustic features and speech synthesis acoustic features from the audio data corresponding to the received source speaker's sentence, uses the speech recognition acoustic features to obtain speech recognition hidden features, and uses the speech synthesis acoustic features to obtain speech synthesis coding features; uses the hidden features and the coding features as intermediaries, uses the timbre conversion model to convert the speech recognition acoustic features corresponding to the source speaker into speech synthesis acoustic features corresponding to the specific target speaker, and then uses the speech synthesis acoustic features to generate the specific target speaker's audio signal. Since multiple acoustic features are used for joint modeling, a better voice changing effect can be obtained; and since the coding features obtained by feature compression of the speech synthesis acoustic features of the entire sentence are added, the rhythm and emotion of the converted sound can be closer to the corresponding sound characteristics of the source speaker.

[0056] Furthermore, in the scheme of the present invention, when modeling, the general voice change model is first trained using the audio data of multiple speakers, and then adaptive training is performed using a small amount of audio data of a specific target speaker based on the general voice change model to obtain a timbre conversion model corresponding to the specific target speaker. Since adaptive training is performed on the audio data of a specific target speaker based on the general voice change model, the parameters of the timbre conversion model obtained by training can be made more accurate, and then the acoustic features of speech synthesis obtained using the timbre conversion model are more consistent with the voice characteristics of the specific target speaker, so that the final synthesized audio signal has a better effect. Moreover, when targeting different specific target speakers, only a small amount of audio data of the specific target speaker needs to be recorded, and there is no need to record the parallel corpus corresponding to the source speaker, which greatly simplifies the collection of training corpus. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0058] Figure 1 is a flow chart of constructing a universal voice changing model in the voice changing method according to an embodiment of the present invention;

[0059] Figure 2 It is a schematic diagram of a topological structure of a universal voice changing model in the voice changing method according to an embodiment of the present invention;

[0060] Figure 3 is a flow chart of a voice changing method according to an embodiment of the present invention;

[0061] Figure 4Schematic diagram of the model training and testing process in the voice changing method according to an embodiment of the present invention;

[0062] Figure 5 is a structural block diagram of a voice changing device according to an embodiment of the present invention;

[0063] Figure 6 is a block diagram of a device for voice changing method according to an exemplary embodiment;

[0064] Figure 7 It is a schematic diagram of the structure of the server in the embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to enable persons skilled in the art to better understand the solutions of the embodiments of the present invention, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and implementation modes.

[0066] The embodiment of the present invention provides a voice changing method and device, which pre-builds a timbre conversion model corresponding to a specific target speaker, extracts speech recognition acoustic features and speech synthesis acoustic features from audio data corresponding to a sentence of a received source speaker, obtains speech recognition hidden features using the speech recognition acoustic features, and obtains speech synthesis coding features using the speech synthesis acoustic features; uses the hidden features and the coding features as intermediaries, uses the timbre conversion model to convert the speech recognition acoustic features corresponding to the source speaker into speech synthesis acoustic features corresponding to the specific target speaker, and then uses the speech synthesis acoustic features to generate an audio signal of the specific target speaker.

[0067] In practical applications, the timbre conversion model can be obtained by collecting a large amount of audio data of specific target speakers for training; or firstly, a universal voice change model is trained using audio data of multiple speakers, and then adaptively trained using a small amount of audio data of specific target speakers on the basis of the universal voice change model to obtain a timbre conversion model corresponding to the specific target speaker.

[0068] like Figure 1 FIG. 1 is a flowchart of constructing a universal voice changing model in the voice changing method according to an embodiment of the present invention, comprising the following steps:

[0069] Step 101: collect audio data of multiple speakers as training data.

[0070] The universal voice changing model is not targeted at a specific target speaker, and thus can be trained based on audio data of multiple speakers.

[0071] Step 102: extracting speech recognition acoustic features and speech synthesis acoustic features from the training data.

[0072] The speech recognition acoustic features may include but are not limited to any one or more of the following: MFCC (Mel-scale Frequency Cepstral Coefficients), PLP (Perceptual Linear Predictive) parameters. The MFCC is a cepstral parameter extracted in the Mel scale frequency domain, and the Mel scale describes the nonlinear characteristics of the human ear frequency; the PLP parameter is a feature parameter based on an auditory model, which is a set of coefficients of a full-pole model prediction polynomial, equivalent to an LPC (Linear Prediction Coefficient) feature. The extraction of the speech recognition acoustic features can adopt existing technologies and will not be described in detail here.

[0073] The speech synthesis acoustic features include: Mel Spectrograms and other features.

[0074] Step 103, using the speech recognition acoustic features to obtain speech recognition latent features.

[0075] The hidden layer feature refers to the output of the hidden layer of the speech recognition model. In the embodiment of the present invention, the speech recognition model can adopt a neural network model, and the neural network model can include one or more hidden layers. Accordingly, the speech recognition acoustic feature is input into the speech recognition model to obtain the output of the hidden layer. In practical applications, one or more hidden layer outputs can be used as the hidden layer features of the speech recognition.

[0076] Step 104: Obtain speech synthesis coding features using the speech synthesis acoustic features.

[0077] Specifically, the speech synthesis acoustic features can be input into a pre-built reference coding model, and the speech synthesis coding features can be obtained according to the output of the reference coding model. The reference coding model can adopt a neural network, such as a multi-layer convolution plus a unidirectional GRU (Gated Recurrent Unit) structure, which is used to compress an audio signal of variable length into a feature vector of fixed length.

[0078] It should be noted that the speech synthesis coding feature is the result of compressing the acoustic features of a complete sentence.

[0079] Step 105, using the speech recognition hidden layer features, speech synthesis coding features and the speech synthesis acoustic features, a universal voice change model is trained.

[0080] The general voice changing model and the timbre conversion model corresponding to the specific target speaker may adopt a neural network model, such as CNN-LSTM (convolutional neural network and long short-term memory network).

[0081] like Figure 2 , which is a schematic diagram of a topological structure of a universal voice changing model in the voice changing method according to an embodiment of the present invention.

[0082] The input of the universal voice changing model includes: speech recognition hidden layer feature A, speech recognition hidden layer feature B, and speech synthesis coding feature obtained by using the source audio speech synthesis acoustic feature through the reference coding model, and the output is the target audio speech synthesis acoustic feature. Among them, the speech recognition hidden layer feature A passes through multiple neural network models such as convolution layer, pooling layer, residual layer, etc. to obtain hidden layer 1 and hidden layer 2; the speech recognition hidden layer feature B passes through multiple layers of DNN to obtain hidden layer 3; the source audio speech synthesis acoustic feature passes through the reference coding model to obtain hidden layer 4; hidden layer 1, hidden layer 2, hidden layer 3 and hidden layer 4 are combined as the input of the LSTM model.

[0083] On the basis of the above-mentioned universal voice changing model, for a specific target speaker, a small amount of audio data of the specific target speaker can be collected, and the universal voice changing model can be adaptively trained using the audio data of the specific target speaker to obtain a timbre conversion model corresponding to the specific target speaker.

[0084] The adaptive training process is similar to the training process of the general voice changing model, except for the difference in training data.

[0085] Since the hidden features output by the recognition acoustic model contain fewer timbre features of the source speaker while retaining semantic information and some prosodic information, the voice change model is used to learn the mapping relationship between the hidden features and the synthetic acoustic features of the target speaker, thereby realizing the conversion of the timbre of the source speaker to the target speaker.

[0086] The voice changing method provided by the embodiment of the present invention utilizes the above-mentioned timbre conversion model to convert the speech recognition acoustic features of the source speaker into the speech synthesis acoustic features of the specific target speaker, and then generates the specific target speaker audio signal according to the speech synthesis acoustic features, thereby realizing real-time conversion of the source speaker audio data into the specific target speaker audio signal.

[0087] like Figure 3 FIG. 1 is a flow chart of a voice changing method according to an embodiment of the present invention, comprising the following steps:

[0088] Step 301: Receive a source speaker's sentence.

[0089] The source speaker's sentence is a complete sentence of the source speaker, that is, the audio corresponding to the complete sentence.

[0090] Step 302: extracting speech recognition acoustic features and speech synthesis acoustic features from the source speaker's sentence.

[0091] Similar to the model training stage, the speech recognition acoustic features may include but are not limited to any one or more of the following: MFCC, PLP, etc.; the speech synthesis acoustic features at least include Mel spectrum.

[0092] Step 303: Obtain speech recognition latent features using the speech recognition acoustic features.

[0093] The speech recognition hidden features can be obtained by inputting the speech recognition acoustic features into a speech recognition model. Specifically, one or more hidden layer outputs in the speech recognition model can be used as the hidden features of the speech recognition.

[0094] Step 304: Obtain speech synthesis coding features using the speech synthesis acoustic features.

[0095] Specifically, the speech synthesis acoustic features may be input into the reference coding model, and the speech synthesis coding features may be obtained according to the output of the reference coding model.

[0096] It should be noted that the speech synthesis coding feature is the result of compressing the acoustic features of a complete sentence.

[0097] Step 305: input the speech recognition latent features and the speech synthesis coding features into a pre-built timbre conversion model corresponding to a specific target speaker to obtain the speech synthesis acoustic features of the specific target speaker.

[0098] The input of the timbre conversion model is the speech recognition hidden layer features obtained in the above step 303 and the speech synthesis coding features obtained in the above step 304, and the output is the speech synthesis acoustic features.

[0099] By using the timbre conversion model, the speech recognition acoustic features of the source speaker can be converted into speech synthesis acoustic features with the voice characteristics of a specific target speaker.

[0100] Step 306: Generate an audio signal of the specific target speaker by using the speech synthesis acoustic features of the specific target speaker.

[0101] Specifically, a neural network vocoder such as wavenet / wavernn can be used to generate a speech signal, and the speech synthesis acoustic features can be synthesized into a speech signal, that is, the transformation of any source speaker's speech into a target speaker's speech can be achieved.

[0102] In order to better understand the present invention, Figure 4A schematic diagram showing the model training and testing process in the voice changing method according to an embodiment of the present invention is shown.

[0103] It should be noted that the type and quantity of specific acoustic features included in the speech synthesis acoustic features extracted from the source speaker's sentence in the above step 302 may be the same as or different from the type and quantity of specific acoustic features included in the speech synthesis acoustic features of the specific target speaker obtained in step 305, and this embodiment of the present invention does not limit this.

[0104] The voice changing method provided by the embodiment of the present invention pre-builds a timbre conversion model corresponding to a specific target speaker, extracts speech recognition acoustic features and speech synthesis acoustic features from the received source speaker's sentence, uses the speech recognition acoustic features to obtain speech recognition hidden features, and uses the speech synthesis acoustic features to obtain speech synthesis coding features; uses the hidden features and the coding features as intermediaries, uses the timbre conversion model to convert the speech recognition acoustic features corresponding to the source speaker into speech synthesis acoustic features corresponding to the specific target speaker, and then uses the speech synthesis acoustic features to generate the specific target speaker's audio signal. Since multiple acoustic features are used for joint modeling, a better voice changing effect can be obtained; and since the coding features obtained by feature compression of the speech synthesis acoustic features of the entire sentence are added, the rhythm and emotion of the converted sound can be closer to the sound characteristics of the source speaker.

[0105] In addition, in the scheme of the present invention, when modeling, the general voice change model is first trained using the audio data of multiple speakers, and then adaptive training is performed using a small amount of audio data of a specific target speaker based on the general voice change model to obtain a timbre conversion model corresponding to the specific target speaker. Since adaptive training is performed on the audio data of a specific target speaker based on the general voice change model, the parameters of the timbre conversion model obtained by training can be made more accurate, and then the acoustic features of speech synthesis obtained using the timbre conversion model are more in line with the voice characteristics of the specific target speaker, so that the final synthesized audio signal has a better effect. Moreover, when targeting different specific target speakers, only a small amount of audio data of the specific target speaker needs to be recorded, and there is no need to record the parallel corpus corresponding to the source speaker, which greatly simplifies the collection of training corpus.

[0106] Accordingly, an embodiment of the present invention further provides a voice changing device, such as Figure 5 Shown is a structural block diagram of the device.

[0107] In this embodiment, the device includes the following modules:

[0108] A receiving module 501 is used to receive a source speaker's sentence;

[0109] A feature extraction module 502, used to extract speech recognition acoustic features and speech synthesis acoustic features from the source speaker's sentence;

[0110] A hidden layer feature acquisition module 503, used to obtain a speech recognition hidden layer feature using the speech recognition acoustic feature;

[0111] A coding feature acquisition module 504, configured to obtain a speech synthesis coding feature using the speech synthesis acoustic feature;

[0112] The feature conversion module 505 is used to input the speech recognition hidden layer features and the speech synthesis coding features into a pre-built timbre conversion model corresponding to a specific target speaker to obtain the speech synthesis acoustic features of the specific target speaker;

[0113] The speech synthesis module 506 is used to generate an audio signal of the specific target speaker by using the speech synthesis acoustic features of the specific target speaker.

[0114] In the embodiment of the present invention, the speech recognition acoustic features may include but are not limited to any one or more of the following: MFCC, PLP, etc.; the speech synthesis acoustic features at least include Mel spectrum.

[0115] The hidden layer feature acquisition module 503 can specifically input the speech recognition acoustic features into a speech recognition model to obtain speech recognition hidden layer features. The speech recognition model can adopt a neural network model, such as LSTM (Long Short-Term Memory), LC-CLDNN (Latency-controlled CLDNN), etc. The CLDNN is a neural network model constructed by using convolution structure, loop structure, and full connection structure at the same time.

[0116] In practical applications, one or more hidden layer outputs in the speech recognition model may be used as hidden layer features of the speech recognition.

[0117] The coding feature acquisition module 504 can specifically input the speech synthesis acoustic feature into a pre-built reference coding model, and obtain the speech synthesis coding feature according to the output of the reference coding model. The specific structure of the reference coding model has been described in detail above and will not be repeated here.

[0118] The feature conversion module 505 inputs the speech recognition hidden layer features and the speech synthesis coding features into a pre-built timbre conversion model corresponding to a specific target speaker to obtain the speech synthesis acoustic features of the specific target speaker. The speech synthesis acoustic features may include features such as Mel spectrum. Accordingly, the speech synthesis module 504 may use a neural network vocoder such as wavenet / wavernn to generate an audio signal of a specific target speaker using these speech synthesis acoustic features, thereby realizing the conversion of any source speaker's speech to the target speaker's speech.

[0119] The voice changing device provided by the embodiment of the present invention pre-builds a timbre conversion model corresponding to a specific target speaker, extracts speech recognition acoustic features and speech synthesis acoustic features from the received source speaker audio data, uses the speech recognition acoustic features to obtain speech recognition hidden features, and uses the speech synthesis acoustic features to obtain speech synthesis coding features; uses the hidden features and the coding features as intermediaries, uses the timbre conversion model to convert the speech recognition acoustic features corresponding to the source speaker into speech synthesis acoustic features corresponding to the specific target speaker, and then uses the speech synthesis acoustic features to generate the specific target speaker audio signal. Since multiple acoustic features are used for joint modeling, a better voice changing effect can be obtained; and since the coding features obtained by feature compression of the speech synthesis acoustic features of the entire sentence are added, the rhythm and emotion of the converted sound can be closer to the sound characteristics of the source speaker.

[0120] In practical applications, the timbre conversion model can be constructed by a corresponding timbre conversion model construction module. The timbre conversion model construction module can be a part of the device of the present invention or can be independent of the device of the present invention, which is not limited.

[0121] The timbre conversion model construction module can specifically obtain the timbre conversion model by collecting a large amount of audio data of a specific target speaker for training, or by using the audio data of multiple speakers to train a universal voice change model, and then adaptively training using a small amount of audio data of a specific target speaker on the basis of the universal voice change model to obtain the timbre conversion model corresponding to the specific target speaker.

[0122] The universal voice changing model can be constructed by a corresponding universal model construction module. Similarly, the universal model construction module can be a part of the device of the present invention or can be independent of the device of the present invention, which is not limited.

[0123] It should be noted that both the training of the general voice changing model and the adaptive training based on the general voice changing model are an iterative calculation process. Therefore, in practical applications, the general model construction module and the timbre conversion model construction module can be combined into one functional module or two independent functional modules, which is not limited. The iterative calculation process of the two is the same, but the training data is different.

[0124] In a specific embodiment, the timbre conversion model building module may include the following units:

[0125] A target data collection unit, used to collect a large amount of audio data of a specific target speaker as training data;

[0126] A feature extraction unit, used to extract speech recognition acoustic features and speech synthesis acoustic features from the training data;

[0127] A hidden layer feature acquisition unit, used to obtain a speech recognition hidden layer feature using the speech recognition acoustic feature;

[0128] A coding feature acquisition unit, used to obtain a speech synthesis coding feature using the speech synthesis acoustic feature;

[0129] The parameter training unit is used to train a timbre conversion model corresponding to the specific target speaker by using the hidden layer features, the speech synthesis coding features and the speech synthesis acoustic features.

[0130] In another specific embodiment, the general model building module may include the following units:

[0131] A universal data collection unit, used for collecting audio data of multiple speakers as training data;

[0132] A feature extraction unit, used to extract speech recognition acoustic features and speech synthesis acoustic features from the training data;

[0133] A hidden layer feature acquisition unit, used to obtain a speech recognition hidden layer feature using the speech recognition acoustic feature;

[0134] A coding feature acquisition unit, used to obtain a speech synthesis coding feature using the speech synthesis acoustic feature;

[0135] The universal parameter training unit is used to train a universal voice changing model by utilizing the speech recognition hidden layer features, the speech synthesis coding features and the speech synthesis acoustic features.

[0136] Accordingly, the timbre conversion model building module may include the following units:

[0137] A target data collection unit, used to collect audio data of a specific target speaker;

[0138] The model training unit is used to adaptively train a universal voice changing model pre-built based on the audio data of multiple speakers using the audio data of the specific target speaker, so as to obtain a timbre conversion model corresponding to the specific target speaker.

[0139] The process of adaptive training mainly involves extracting speech recognition acoustic features and speech synthesis acoustic features from the audio data of the specific target speaker, and using the speech recognition acoustic features and the speech synthesis acoustic features to obtain speech recognition latent features and speech synthesis coding features respectively, and then using the latent features, coding features and speech synthesis acoustic features to train a timbre conversion model corresponding to the specific target speaker through iterative calculation.

[0140] By using the solution of this embodiment, a timbre conversion model corresponding to the specific target speaker can be obtained by collecting a small amount of audio data of a specific target speaker and adaptively training based on a universal voice change model, so that the parameters of the trained timbre conversion model can be more accurate, and then the speech synthesis acoustic features obtained by using the timbre conversion model are more consistent with the voice characteristics of the specific target speaker, so that the final synthesized audio signal has a better effect. Moreover, when targeting different specific target speakers, only a small amount of audio data of the specific target speaker needs to be recorded, and there is no need to record the parallel corpus corresponding to the source speaker, which greatly simplifies the collection of training corpus.

[0141] Figure 6 8 is a block diagram of a device 800 for a voice changing method according to an exemplary embodiment. For example, the device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0142] Reference Figure 6 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0143] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0144] The memory 804 is configured to store various categories of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any category of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0145] The power component 806 provides power to the various components of the device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0146] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0147] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0148] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0149] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0150] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0151] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0152] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to complete the above-mentioned key mis-touch error correction method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0153] The present invention also provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal can execute all or part of the steps in the above-mentioned method embodiment of the present invention.

[0154] Figure 7 1 is a schematic diagram of the structure of the server in an embodiment of the present invention. The server 1900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1922 (for example, one or more processors) and memory 1932, and one or more storage media 1930 (for example, one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be short-term storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1922 can be configured to communicate with the storage medium 1930 and execute a series of instruction operations in the storage medium 1930 on the server 1900.

[0155] The server 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958, one or more keyboards 1956, and / or, one or more operating systems 1941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0156] Those skilled in the art will readily appreciate other embodiments of the invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0157] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0158] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A voice changing method, characterized in that: The method comprises: receiving a source speaker sentence, wherein the source speaker sentence is a complete sentence of the source speaker; Extracting speech recognition acoustic features and speech synthesis acoustic features from the source speaker's sentence; Obtaining speech recognition latent features using the speech recognition acoustic features; Using the speech synthesis acoustic features to obtain speech synthesis coding features, wherein the speech synthesis coding features are the result of compressing the complete sentence; Inputting the speech recognition latent features and the speech synthesis coding features into a pre-built timbre conversion model corresponding to a specific target speaker to obtain the speech synthesis acoustic features of the specific target speaker; An audio signal of the specific target speaker is generated by using the speech synthesis acoustic features of the specific target speaker.

2. The method according to claim 1, characterized in that The method further comprises constructing the timbre conversion model corresponding to the specific target speaker in the following manner: Collect audio data from a specific target speaker; The audio data of the specific target speaker is used to adaptively train a universal voice change model pre-constructed based on the audio data of multiple speakers to obtain a timbre conversion model corresponding to the specific target speaker.

3. The method according to claim 2, characterized in that The method further includes: constructing a universal voice changing model based on the audio data of multiple speakers, specifically including: Collect audio data from multiple speakers as training data; Extracting speech recognition acoustic features and speech synthesis acoustic features from the training data; Obtaining speech recognition latent features using the speech recognition acoustic features; Obtaining speech synthesis coding features using the speech synthesis acoustic features; The universal voice change model is trained by utilizing the speech recognition hidden layer features, speech synthesis coding features and the speech synthesis acoustic features.

4. The method according to claim 1, characterized in that The method of obtaining the speech recognition hidden layer features by using the speech recognition acoustic features comprises: The speech recognition acoustic features are input into a speech recognition model to obtain speech recognition latent features.

5. The method according to claim 1, characterized in that The step of obtaining the speech synthesis coding feature by using the speech synthesis acoustic feature comprises: The speech synthesis acoustic features are input into a pre-built reference coding model to obtain speech synthesis coding features.

6. The method according to claim 1, characterized in that The speech recognition acoustic features include any one or more of the following: Mel-frequency cepstral coefficients, perceptual linear prediction parameters.

7. The method according to claim 1, characterized in that The speech synthesis acoustic features include: Mel spectrum.

8. A voice changing device, characterized in that: The device comprises: A receiving module, configured to receive a sentence of a source speaker, wherein the sentence of the source speaker is a complete sentence of the source speaker; A feature extraction module, used to extract speech recognition acoustic features and speech synthesis acoustic features from the source speaker's sentence; A hidden layer feature acquisition module, used to obtain speech recognition hidden layer features using the speech recognition acoustic features; A coding feature acquisition module, used to obtain a speech synthesis coding feature using the speech synthesis acoustic feature, wherein the speech synthesis coding feature is a result of compressing the complete sentence; A feature conversion module, used for inputting the speech recognition hidden layer features and the speech synthesis coding features into a pre-built timbre conversion model corresponding to a specific target speaker, to obtain the speech synthesis acoustic features of the specific target speaker; The speech synthesis module is used to generate an audio signal of the specific target speaker by using the speech synthesis acoustic features of the specific target speaker.

9. An electronic device, characterized in that: include: One or more processors, memory; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the method according to any one of claims 1 to 7.

10. A readable storage medium having instructions stored thereon, wherein the instructions are executed to implement the method according to any one of claims 1 to 7.

Images