Voice cloning method and device based on neural network model, equipment and medium

Abstract

The invention discloses a voice cloning method and device based on a neural network model, equipment and a medium. The method comprises the following steps: performing audio information extraction on cloned reference voice to obtain a voice signal and background noise; extracting corresponding voice features from the voice signal through a preset feature extraction mode, and performing noise extraction on the background noise to obtain background voice features; performing weighted fusion on the voice features and the background voice features to obtain target fusion features; and according to the target fusion feature and a preset to-be-synthesized text, voice cloning is carried out through a trained target neural network model, and target cloned voice is generated. The voice cloning method and device can be applied to voice cloning in businesses such as financial insurance and medical health, and can solve the problem that voice cloning cannot be efficiently carried out in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of voice cloning and can be applied to fields such as finance, insurance, and medical health. In particular, it relates to a voice cloning method, device, equipment, and medium based on a neural network model. Background Art

[0002] With the advancement of technology, artificial intelligence has made significant progress in speech processing. Voice cloning technology, in particular, demonstrates tremendous potential in a variety of scenarios, such as digital human interaction, film and television dubbing, and audiobook production. It also has applications in finance and healthcare, such as personalized intelligent customer service and health education audio. However, existing voice cloning technology faces numerous challenges in practical application, including relying on large amounts of manually annotated data to extract speech features, resulting in high costs and limited data quality. Furthermore, the technology struggles to efficiently decouple and fuse voiceprints, paralinguistic information, and environmental features, lacking a collaborative mechanism. This results in the inability of existing technologies to efficiently perform voice cloning, making it difficult to meet user demands for a high-fidelity, immersive voice experience, and limiting the application and promotion of voice cloning technology in a wider range of scenarios. Summary of the Invention

[0003] The embodiments of the present invention provide a voice cloning method, apparatus, device and medium based on a neural network model, aiming to solve the problem that voice cloning cannot be performed efficiently in the prior art.

[0004] In a first aspect, an embodiment of the present invention provides a speech cloning method based on a neural network model, comprising: extracting audio information from a cloned reference speech to obtain a speech signal and background noise; extracting corresponding speech features from the speech signal using a preset feature extraction method, and performing noise extraction on the background noise to obtain background sound features; performing weighted fusion on the speech features and the background sound features to obtain target fusion features; and performing speech cloning based on the target fusion features and a preset text to be synthesized using a trained target neural network model to generate a target cloned speech.

[0005] In a second aspect, an embodiment of the present invention further provides a speech cloning device based on a neural network model, comprising: an extraction unit for extracting audio information from a cloned reference speech to obtain a speech signal and background noise; an acquisition unit for extracting corresponding speech features from the speech signal using a preset feature extraction method, and performing noise extraction on the background noise to obtain background sound features; a fusion unit for performing weighted fusion on the speech features and the background sound features to obtain target fusion features; and a cloning unit for performing speech cloning based on the target fusion features and a preset text to be synthesized using a trained target neural network model to generate a target cloned speech.

[0006] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0007] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the above method can be implemented.

[0008] Embodiments of the present invention provide a method, apparatus, device, and medium for voice cloning based on a neural network model. The method includes: extracting audio information from a cloned reference voice to obtain a voice signal and background noise; extracting corresponding voice features from the voice signal using a preset feature extraction method, and performing noise extraction on the background noise to obtain background sound features; weightedly fusing the voice features and background sound features to obtain target fused features; and performing voice cloning based on the target fused features and a preset text to be synthesized using a trained target neural network model to generate a target cloned voice. The embodiment of the present invention extracts audio information from the cloned reference voice to separate the voice signal and background noise from the original audio, laying the foundation for subsequent feature extraction. The voice features and background noise of the voice signal are extracted to obtain information such as the speaker's timbre, intonation, and background noise, thereby enhancing the realism of the cloned voice. The extracted speech features are weightedly fused with the background noise features to generate target fusion features. Voice cloning is performed based on the target fusion features and the preset text to be synthesized to obtain the target cloned voice. While achieving automatic conversion from text to speech, the interaction between voice and background noise in a real environment is simulated, making the cloned voice closer to the actual scene. Voice cloning can be performed efficiently, so that users' needs for high-fidelity and immersive voice experience can be met in fields such as medicine and finance. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 A schematic diagram of a flow chart of a voice cloning method based on a neural network model provided in an embodiment of the present invention;

[0011] Figure 2 A schematic diagram of a sub-process of a voice cloning method based on a neural network model provided by an embodiment of the present invention;

[0012] Figure 3 A schematic diagram of a sub-process of a voice cloning method based on a neural network model provided by an embodiment of the present invention;

[0013] Figure 4 A schematic diagram of a sub-process of a voice cloning method based on a neural network model provided by an embodiment of the present invention;

[0014] Figure 5 A schematic diagram of a sub-process of a voice cloning method based on a neural network model provided by an embodiment of the present invention;

[0015] Figure 6 A schematic diagram of a sub-process of a voice cloning method based on a neural network model provided by an embodiment of the present invention;

[0016] Figure 7 A schematic diagram of a sub-process of a voice cloning method based on a neural network model provided by an embodiment of the present invention;

[0017] Figure 8 A schematic block diagram of a voice cloning device based on a neural network model provided by an embodiment of the present invention;

[0018] Figure 9 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0020] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0021] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in the specification and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0022] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0023] See also Figure 1 , Figure 1 A flowchart of a voice cloning method based on a neural network model provided in an embodiment of the present invention. The voice cloning method based on a neural network model in this embodiment can be applied to scenarios where medical and health management and personalized financial services are integrated. For example, customized voice assistants can be generated for elderly patients or those with speech disorders in remote medical consultations, while creating exclusive voice customer service images for high-net-worth clients in financial management services. By adopting this method, personalized adaptation of professional diagnosis and treatment voices in medical scenarios can be achieved to improve patient compliance; in the financial field, by cloning the voice style of account managers, the voice cloning can be performed efficiently while meeting the user's demand for high-fidelity and immersive voice experience.

[0024] Figure 1 FIG. 1 is a flow chart of a voice cloning method based on a neural network model provided by an embodiment of the present invention. As shown in the figure, the method includes the following steps S110-S140.

[0025] S110: extracting audio information from the cloned reference speech to obtain speech signals and background noise.

[0026] In this embodiment, the cloned reference voice is an original audio file (such as a recording of a doctor or customer service representative) that serves as a voice cloning template. The voice signal refers to the human voice, including biometric features such as pitch, timbre, and speaking rate. The background noise is components other than the target human voice, such as background sounds. Audio information is extracted from the cloned reference voice to obtain the voice signal and background noise. Specifically, the cloned reference voice can be preprocessed, for example, by segmenting the audio into short time frames (e.g., 20-40ms) to increase the speed of voice cloning. It is important to note that the duration of the cloned reference voice should be no less than 15 seconds to ensure that sufficient paralinguistic information describing the speaker's speaking habits can be extracted from this reference voice. The cloned reference voice can be subjected to speech separation based on spectral analysis to obtain the voice signal and background noise, or a deep learning model (e.g., Wave-U-Net) can be trained to learn time-frequency masks of the voice and noise, thereby achieving end-to-end separation to obtain the voice signal and background noise. By extracting the voice signal and background noise from the cloned reference voice, reference data is provided for subsequent voice cloning.

[0027] In one embodiment, if Figure 2 As shown, step S110 includes steps S111-S113.

[0028] S111, generating a speech mask and a noise mask from the cloned reference speech through a preset deep neural network;

[0029] S112, reconstructing the voice mask signal to obtain the voice signal;

[0030] S113: Perform inverse transformation on the noise mask to obtain the background noise.

[0031] In this embodiment, the preset deep neural network is a deep neural network (DNN) designed specifically for speech separation, which predicts speech and noise masks from mixed audio through training and learning. The speech mask and noise mask are weight matrices in the time-frequency domain, which identify the energy distribution of the speech signal or background noise. The cloned reference speech is used to generate a speech mask and a noise mask through a preset deep neural network. Specifically, the input mixed speech is first subjected to a short-time Fourier transform (STFT) to convert it from the time domain to the frequency domain to obtain spectral features. The spectral features are input into the deep neural network, and the network outputs two masks, one is a speech mask and the other is a noise mask. For example, for each time-frequency unit in the frequency domain, a value of the speech mask close to 1 indicates that the unit is mainly a speech component, and a value close to 0 indicates that it is mainly a noise component; the opposite is true for the noise mask. The resulting speech mask is multiplied by the spectrum of the reference speech to produce a spectral estimate of the speech. This is then converted back to the time domain using an inverse short-time Fourier transform (ISTFT), yielding the separated speech signal. Similarly, the background noise signal is obtained by multiplying the noise mask by the spectrum of the reference speech and then inversely transforming it. Feature separation using a pre-set deep neural network allows for efficient extraction of pure speech signals and background noise, providing a high-quality data foundation for subsequent speech cloning.

[0032] S120: Extract corresponding speech features from the speech signal using a preset feature extraction method, perform noise extraction on the background noise, and obtain background sound features.

[0033] In this embodiment, the speech features are parameters extracted from the speech signal that characterize speech characteristics, such as Mel-Frequency Cepstral Coefficients (MFCCs), fundamental frequency (F0), and energy spectrum. The background sound features are parameters extracted from the background noise that characterize noise characteristics, such as noise power spectral density and frequency band energy distribution. The speech signal is subjected to a preset feature extraction method to extract corresponding speech features. Specifically, a model for extracting speech features can be pre-set, and feature extraction is performed based on the preset model. For example, acoustic features can be extracted using a pre-trained acoustic feature extraction model (e.g., wav2vec), where the acoustic features are one of the speech features. The background noise is subjected to noise extraction to obtain background sound features. Specifically, the energy contribution of noise in different frequency bands is calculated (e.g., a high contribution of low-frequency noise may indicate air conditioning noise). A machine learning model (e.g., CNN) is used to determine the noise category (e.g., white noise, impulse noise). Data describing the randomness of the noise, such as mean, variance, and zero-crossing rate, is extracted. Based on this data, the machine learning model outputs a noise feature vector (e.g., frequency band energy distribution + noise type label). By obtaining voice features and background sound features, accurate voice cloning can be performed later.

[0034] In one embodiment, if Figure 3 As shown, step S120 includes steps S121-S122.

[0035] S121. Extract features from the speech signal using a preset acoustic extraction model, a preset paralanguage extraction model, and a preset voiceprint extraction model to obtain acoustic features, paralanguage features, and voiceprint features;

[0036] S121. Determine the speech feature according to the acoustic feature, the paralinguistic feature, and the voiceprint feature.

[0037] In this embodiment, the acoustic extraction refers to parameters describing the physical characteristics of speech, such as pitch, energy, and spectral distribution, which reflect the physical properties of speech. The paralinguistic features are parameters describing non-linguistic information in speech, such as stress, prolongation, breathing, pauses, and laughter. The voiceprint features are biometric features describing the speaker's identity, such as vocal tract shape and pronunciation habits. The speech signal is subjected to feature extraction using a preset acoustic extraction model, a preset paralinguistic extraction model, and a preset voiceprint extraction model to obtain acoustic features, paralinguistic features, and voiceprint features. Specifically, a pre-trained acoustic extraction model (wav2vec) is used to extract corresponding acoustic features based on the mel-spectrogram of the speech signal. Voiceprint features are extracted using a pre-trained model for extracting the speaker's voiceprint vector (such as an x-vector embedding model). Voiceprint features such as speaking rate, pause duration, and stress position are extracted using a pre-trained pre-trained paralinguistic extraction model (such as an LSTM-based speech rate analysis model). For example, when cloning the voice of a financial customer service representative, we extract acoustic features such as the fundamental frequency and energy spectrum of the customer service representative, extract their voiceprint embedding (e.g., x-vector), and extract paralinguistic features such as speech rate and pause patterns. Based on these acoustic, paralinguistic, and voiceprint features, we determine the customer service representative's voice characteristics. Based on these extracted voice features, we perform voice cloning while taking into account naturalness, identity consistency, and emotional expression.

[0038] In one embodiment, if Figure 4 As shown, the step S122 also includes steps S1221-S122.

[0039] S1221: Train a preset extraction model based on preset language training data to obtain the preset paralanguage extraction model capable of identifying a paralanguage;

[0040] S1222: Locate and extract the paralanguage in the speech signal according to the preset paralanguage extraction model to obtain the paralanguage features.

[0041] In this embodiment, the preset language training data is used to train a paralanguage extraction model, and is labeled with paralanguage information (such as laughter, sighs, and pauses). Before training the preset extraction model based on the preset language training data and obtaining the preset paralanguage extraction model capable of identifying paralanguage, training data is collected and labeled. For example, in a financial context, customer service recordings may be labeled with "laughter" (positive emotion) and "long pauses" (hesitation); in a medical context, doctor consultation recordings may be labeled with "sighs" (fatigue) and "accelerated speech" (urgency). A suitable model architecture is selected as the preset extraction model. For example, a CNN (Convolutional Neural Network) is suitable for extracting paralanguage patterns from spectrograms and is used as the preset extraction model. The preset extraction model is trained based on the labeled preset language training data to minimize the prediction error of paralanguage labels, thereby obtaining the preset paralanguage extraction model. The pre-set paralanguage extraction model is used to locate and extract paralanguage from the speech signal. Specifically, the trained model analyzes the input speech frame by frame, marking the temporal position of paralanguage events (e.g., laughter at the 2.3th second mark) for localization. Feature extraction is then performed based on this localization information, outputting information such as the start and end times, probability, and frequency of paralanguage events to obtain paralanguage features. By training a dedicated paralanguage extraction model, key paralanguage features in financial and medical scenarios can be accurately located and extracted, facilitating sentiment analysis and voice cloning.

[0042] S130: Perform weighted fusion on the speech features and the background sound features to obtain target fusion features.

[0043] In this embodiment, the target fusion feature refers to a comprehensive feature that combines the speech features and the background sound features. The speech features and background sound features are weighted and fused to obtain the target fusion feature. Specifically, the speech features (e.g., MFCC) and background sound features (e.g., noise energy spectrum) can be pre-normalized using Z-scores to eliminate scale differences between the speech and background sound features. Finally, the two features are weighted and fused according to preset weights. The speech features can be weighted higher (e.g., 0.7-0.9) to ensure that the cloned voice retains the speaker's identity and style. The background sound features can be weighted lower (e.g., 0.1-0.3) to fine-tune the cloned voice to the target environment. For example, in a medical consultation scenario where environmental authenticity must be preserved (e.g., simulated office noise), the speech features are weighted at 0.8 and the background sound features are weighted at 0.2. For a financial customer service voice cloning scenario, the background noise of the customer service office (e.g., keyboard sounds, air conditioning) is extracted and the background sound features are weighted at 0.1 to simulate a realistic service environment. The weighted fusion process is: Ftarget = a*Fvoice + b*Fnoise, where Ftarget is the target fusion feature, a and b are the weights of the voice and background noise features, respectively, and Fvoice and Fnoise are the normalized voice and background noise features, respectively. By fusing these voice and background noise features to create weighted features, the cloned voice is rendered more natural, preserving subtle ambient sounds and enhancing the user's sense of authenticity.

[0044] S140 , performing voice cloning based on the target fusion features and the preset text to be synthesized through a trained target neural network model to generate a target cloned voice.

[0045] In this embodiment, the preset text to be synthesized is text content that needs to be converted into speech, such as financial customer service scripts or doctor's consultation statements. The target neural network model is a trained deep learning model (such as F5-TTS) that maps text and features into synthesized speech. The target fused features and the preset text to be synthesized are input into the target neural network model. The network model generates a mel-spectrogram based on the input content and converts the mel-spectrogram into a time-domain waveform to obtain the target cloned speech. For example, the target fused features have a speech feature weight of 0.9 (high clarity) and a background sound feature weight of 0.1 (mild keyboard sounds). The preset text to be synthesized is: "Your loan application has been approved with an interest rate of 4.5%." The model generates clear, professional speech while retaining mild office environment sounds, enhancing customer trust. By combining the target fused features with the text input, the target neural network model can generate high-quality cloned speech that meets user needs for a high-fidelity, immersive speech experience.

[0046] In one embodiment, if Figure 5As shown, the step S140 also includes steps S1401-S1403 before the step S140.

[0047] S1401, standardizing the collected audio training data, and removing background sound from the processed audio training data using a preset removal algorithm;

[0048] S1402: Perform speaker and voice-to-text recognition on the audio training data after background sound removal to obtain target training data;

[0049] S1403: Train the neural network model according to the target training data to obtain the target neural network model.

[0050] In this embodiment, the audio training data is audio data collected from various data sources, such as doctor consultation audio, financial customer service voice, etc. The preset removal algorithm is an algorithm for suppressing or separating background noise, such as spectral subtraction, deep learning denoising model, etc. The collected audio training data is standardized, and the processed audio training data is subjected to a preset removal algorithm to remove background sound. Specifically, the collected audio training data is standardized according to preset standard data such as sampling rate, bit depth (such as 16-bit), and mono, so that the format of all audio is consistent. Speech and noise are separated by a pre-trained UNet or CRN model or spectral subtraction to remove background sound. It should be noted that the background sound is removed in order to more conveniently extract text and other content in the audio later. The audio training data, after background noise removal, is subjected to speaker and speech-to-text recognition to obtain target training data. Specifically, a speaker verification model (such as VGGVox) is used to extract speaker embeddings to identify the speaker. A speech recognition model, such as an ASR model (such as DeepSpeech), is then used to convert the speech into text, thereby obtaining the target training data. The target training data includes speaker information, speech-to-text, and standardized audio training data including background noise. For example, in the financial sector, the target training data may include customer service recordings annotated with the speaker ID (such as "Customer Service_Zhang") and the text of the dialogue (such as "What service do you need?"). The neural network model is trained based on the target training data to obtain the target neural network model. The specific training process is not limited, and the difference between the cloned speech and the real speech should be minimized. By obtaining target training data through standardization, background noise removal, and speaker and text recognition, the target neural network model ultimately trained can achieve high-quality speech cloning, meeting the requirements of professionalism, compliance, and naturalness in the financial and medical fields.

[0051] In one embodiment, if Figure 6 As shown, step S1402 also includes steps S14021-S14023.

[0052] S14021. Extracting the voices of all speakers in the audio training data, and merging the voices of the same speaker;

[0053] S14022. Recognize text in speech using a preset speech recognition model to generate primary audio training data with the speaker and text content;

[0054] S14023. Filter the primary audio training data according to a preset voice quality range to obtain the target training data.

[0055] In this embodiment, the speech recognition model is an automatic speech recognition model (such as DeepSpeech, Wav2Vec2), which is used to convert speech into text. In this embodiment, three different speech recognition models are used to recognize text in speech. The speech of all speakers in the audio training data is extracted, and the speech of the same speaker is merged. Specifically, a speaker verification model (such as VGGVox) is used to extract speaker embedding to identify the speaker and splice all speech segments of the same speaker in chronological order. The text in the speech is recognized by a preset speech recognition model to generate primary audio training data with speaker and text content. Specifically, three different speech recognition models are used to recognize the text in the speech, and the merged speech is converted into text to obtain the primary audio training data, wherein the primary audio training data may include three segments of recognized text. The primary audio training data is screened according to a preset voice quality range to obtain the target training data. Specifically, a tool (such as PESQ) is used to evaluate the voice quality and eliminate low-quality segments. For example, segments with a signal-to-noise ratio of <20dB (such as noise recordings caused by microphone failure) are eliminated, and training data with a high text overlap rate is screened out based on the text overlap rate of different speech recognition models. It can be understood that the higher the text overlap rate, the better the audio quality. After layers of screening, the target training data can be obtained. In order to facilitate audio processing, the audio can also be divided into short sentences suitable for training. Through speaker merging, speech recognition and quality screening, the target training data finally obtained can effectively improve the naturalness, professionalism and scene adaptability of the voice cloning model.

[0056] In one embodiment, if Figure 7 As shown, the step S140 also includes steps S141-S143.

[0057] S141, preprocessing and encoding the preset text to be synthesized to obtain a text feature vector;

[0058] S142, fusing the text feature vector with the target fusion feature through the target neural network model to generate an acoustic feature;

[0059] S143: Generate a mel spectrum using a preset generator according to the acoustic features, and convert the mel spectrum into the target cloned speech using a preset vocoder.

[0060] In this embodiment, the target neural network model is a model for fusing text features and target fusion features, and in this embodiment, it is an F5-TTS model (an open source text-to-speech conversion system based on deep learning). The preset text to be synthesized is preprocessed and encoded to obtain a text feature vector. Specifically, the text is converted into a phoneme sequence. For example, the financial text "Please provide your ID number" is converted into a phoneme sequence: [qǐng, tí, gōng, shēn, fèn, zhèng, hào]. After unifying the number and date formats, a pre-trained text encoder (such as BERT) is used to generate a text feature vector. The text feature vector and the target fusion feature are input into the target neural network model for fusion. Acoustic features are generated in the F5-TTS model, and a generator in the model generates a mel spectrum based on the acoustic features, wherein the mel spectrum can reflect the frequency domain characteristics of the speech. The mel spectrum is converted into a time domain waveform through a preset vocoder (such as HiFi-GAN), and the target cloned speech can be obtained based on the time domain waveform. Through text preprocessing, feature fusion, acoustic feature generation and waveform conversion, voice cloning is efficiently performed to meet users' needs for high-fidelity and immersive voice experience.

[0061] Figure 8 FIG is a schematic block diagram of a speech cloning device 200 based on a neural network model provided by an embodiment of the present invention. Figure 8 As shown, corresponding to the above-mentioned voice cloning method based on a neural network model, the present invention also provides a voice cloning device based on a neural network model. The voice cloning device based on a neural network model includes a unit for executing the above-mentioned voice cloning method based on a neural network model. The device can be configured in a terminal such as a desktop computer, a tablet computer, or a laptop computer. Specifically, please refer to Figure 8 The speech cloning device based on the neural network model includes an extraction unit 210, an acquisition unit 220, a fusion unit 230 and a cloning unit 240.

[0062] The extraction unit 210 is configured to extract audio information from the cloned reference speech to obtain a speech signal and background noise.

[0063] In one embodiment, the extraction unit 210 includes a mask unit, a reconstruction unit, and a transformation unit.

[0064] A masking unit, configured to generate a speech mask and a noise mask from the cloned reference speech through a preset deep neural network;

[0065] A reconstruction unit, configured to reconstruct the voice mask signal to obtain the voice signal;

[0066] The transform unit is configured to perform an inverse transform on the noise mask to obtain the background noise.

[0067] The acquisition unit 220 is configured to extract corresponding speech features from the speech signal using a preset feature extraction method, and perform noise extraction on the background noise to obtain background sound features.

[0068] In one embodiment, the acquisition unit 220 includes a speech feature extraction unit, a feature determination unit

[0069] a speech feature extraction unit, configured to extract features from the speech signal using a preset acoustic extraction model, a preset paralinguistic extraction model, and a preset voiceprint extraction model to obtain acoustic features, paralinguistic features, and voiceprint features;

[0070] A feature determination unit is configured to determine the speech feature based on the acoustic feature, the paralinguistic feature, and the voiceprint feature.

[0071] In one embodiment, the acquisition unit 220 includes a paralanguage extraction unit and a positioning unit.

[0072] a paralanguage extraction unit, configured to train a preset extraction model based on preset language training data to obtain the preset paralanguage extraction model capable of identifying the paralanguage;

[0073] A positioning unit is configured to locate and extract the paralanguage in the speech signal according to the preset paralanguage extraction model to obtain the paralanguage feature.

[0074] The fusion unit 230 is configured to perform weighted fusion on the speech feature and the background sound feature to obtain a target fusion feature.

[0075] The cloning unit 240 is configured to perform speech cloning based on the target fusion features and the preset text to be synthesized through a trained target neural network model to generate a target cloned speech.

[0076] In one embodiment, the cloning unit 240 includes a removal unit, a recognition unit, and a training unit.

[0077] A removal unit is used to standardize the collected audio training data and remove background sound from the processed audio training data using a preset removal algorithm;

[0078] The recognition unit is used to perform speaker and voice text recognition on the audio training data after removing the background sound to obtain target training data;

[0079] A training unit is used to train the neural network model according to the target training data to obtain the target neural network model.

[0080] In one embodiment, the cloning unit 240 includes a merging unit, a generating unit, a screening unit

[0081] a merging unit, configured to extract the speech of all speakers in the audio training data and merge the speech of the same speaker;

[0082] A generation unit, configured to recognize text in speech using a preset speech recognition model and generate primary audio training data having a speaker and text content;

[0083] The screening unit is used to screen the primary audio training data according to a preset voice quality range to obtain the target training data.

[0084] In one embodiment, the cloning unit 240 includes

[0085] An encoding unit, configured to preprocess and encode the preset text to be synthesized to obtain a text feature vector;

[0086] A model fusion unit, configured to fuse the text feature vector with the target fusion feature through the target neural network model to generate an acoustic feature;

[0087] The conversion unit is configured to generate a Mel spectrum through a preset generator according to the acoustic features, and convert the Mel spectrum into the target cloned speech through a preset vocoder.

[0088] It should be noted that those skilled in the art will clearly understand that the specific implementation process of the above-mentioned neural network model-based voice cloning device 200 and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.

[0089] The above-mentioned speech cloning device based on the neural network model can be implemented in the form of a computer program. The computer program can be used in Figure 9 Runs on the computer equipment shown.

[0090] See also Figure 9 , Figure 9 This is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 can be a terminal or a server. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, personal digital assistant, wearable device, or other electronic device with communication capabilities. The server can be a standalone server or a server cluster consisting of multiple servers.

[0091] See Figure 9 The computer device 500 includes a processor 502 , a memory, and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0092] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, which, when executed, can enable the processor 502 to perform a voice cloning method based on a neural network model.

[0093] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500.

[0094] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a voice cloning method based on a neural network model.

[0095] The network interface 505 is used to communicate with other devices over the network. Figure 9 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0096] The processor 502 is configured to run a computer program 5032 stored in the memory to implement the steps of the above method.

[0097] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0098] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program includes program instructions, which can be stored in a storage medium that is computer-readable. The program instructions are executed by at least one processor in the computer system to implement the steps in the method of the above-described embodiment.

[0099] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor performs the steps of the above method.

[0100] The storage medium may be any computer-readable storage medium that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

[0101] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0102] In the several embodiments provided herein, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the various units is merely a logical functional division, and actual implementation may employ other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be omitted or not implemented.

[0103] The steps in the methods of the embodiments of the present invention may be adjusted in order, combined, or deleted as needed. The units in the devices of the embodiments of the present invention may be combined, divided, or deleted as needed. Furthermore, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0104] If this integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present invention.

[0105] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A voice cloning method based on a neural network model, characterized in that: include: Extract audio information from the cloned reference speech to obtain speech signals and background noise; Extracting corresponding speech features from the speech signal through a preset feature extraction method, performing noise extraction on the background noise to obtain background sound features; Performing weighted fusion on the speech features and the background sound features to obtain target fusion features; According to the target fusion features and the preset text to be synthesized, voice cloning is performed through a trained target neural network model to generate a target cloned voice.

2. The method according to claim 1, characterized in that The step of extracting audio information from the cloned reference voice to obtain the voice signal and background noise includes: The cloned reference speech is passed through a preset deep neural network to generate a speech mask and a noise mask; Reconstructing the voice mask signal to obtain the voice signal; The noise mask is inversely transformed to obtain the background noise.

3. The method according to claim 1, characterized in that The step of extracting corresponding speech features from the speech signal using a preset feature extraction method includes: Extracting features from the speech signal using a preset acoustic extraction model, a preset paralanguage extraction model, and a preset voiceprint extraction model to obtain acoustic features, paralanguage features, and voiceprint features; The speech feature is determined according to the acoustic feature, the paralinguistic feature, and the voiceprint feature.

4. The method according to claim 3, characterized in that The step of extracting features from the speech signal through a preset paralanguage extraction model to obtain paralanguage features includes: Training a preset extraction model based on preset language training data to obtain the preset paralanguage extraction model capable of identifying the paralanguage; The paralanguage in the speech signal is located and extracted according to the preset paralanguage extraction model to obtain the paralanguage features.

5. The method according to claim 1, wherein Before the step of performing voice cloning based on the target fusion feature and the preset text to be synthesized through the trained target neural network model to generate the target cloned voice, the method includes: The collected audio training data is standardized and the background sound is removed from the processed audio training data using a preset removal algorithm; Perform speaker and speech recognition on the audio training data after removing background sound to obtain target training data; The neural network model is trained according to the target training data to obtain the target neural network model.

6. The method according to claim 5, characterized in that The step of performing speaker and speech recognition on the audio training data after removing the background sound to obtain target training data includes: Extracting the speech of all speakers in the audio training data and merging the speech of the same speaker; Recognize text in speech using a preset speech recognition model to generate primary audio training data with speakers and text content; The primary audio training data is screened according to a preset voice quality range to obtain the target training data.

7. The method according to claim 1, characterized in that The step of performing voice cloning based on the target fusion feature and the preset text to be synthesized through a trained target neural network model to generate a target cloned voice includes: Preprocessing and encoding the preset text to be synthesized to obtain a text feature vector; fusing the text feature vector with the target fusion feature through the target neural network model to generate an acoustic feature; A Mel spectrum is generated by a preset generator according to the acoustic features, and the Mel spectrum is converted into the target clone voice by a preset vocoder.

8. A voice cloning device based on a neural network model, characterized in that: include: An extraction unit, configured to extract audio information from the cloned reference speech to obtain speech signals and background noise; An acquisition unit, configured to extract corresponding speech features from the speech signal using a preset feature extraction method, and perform noise extraction on the background noise to obtain background sound features; A fusion unit, configured to perform weighted fusion on the speech feature and the background sound feature to obtain a target fusion feature; The cloning unit is used to perform voice cloning based on the target fusion features and the preset text to be synthesized through a trained target neural network model to generate a target cloned voice.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 7 can be implemented.

Images