Target voice generation method and system

Abstract

The invention discloses a target voice generation method and system. The method comprises the following steps: acquiring a first voice, a second voice and a target text; extracting information features according to the first voice; extracting style features according to the second voice; extracting phoneme corresponding features according to the target text; and generating a target voice according to the information feature, the style feature and the phoneme corresponding feature. According to the invention, on the basis of the information feature of the first voice, the style feature of the second voice is further integrated; even if the training samples are limited, the audio of any speaker in diversified scenes can be flexibly generated, so that the dimension and depth of audio expression are greatly enriched; moreover, the fusion of the cross-speech features remarkably improves the expressive power and persuasion of the target speech, enables the target speech to be closer to the emotion concentration of real human communication, and can greatly improve the naturalness of the target speech, thereby causing the resonance of audiences.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a target speech generation method and system. Background Art

[0002] With the rapid development of artificial intelligence technology, debate systems, as an important branch of natural language processing and intelligent interaction, are gradually moving from theoretical research to practical applications.

[0003] At present, debate systems such as "Debater" have emerged. These systems use complex algorithms and big data training to extract features from existing debate corpora, and then simulate and generate voices with similar debate styles and content. This technology uses a large amount of debate voice data of a specific individual as the training basis to accurately capture and reproduce the voice features of the individual.

[0004] However, although the existing technology can generate corresponding speech based on the learned features, its output is often limited by the individual features contained in the training data, that is, it can only generate debate speech similar to the individual voice and style in the training set. When the target speaker does not have enough debate speech data for training, the target speaker's debate speech cannot be generated. Summary of the invention

[0005] In order to solve the above problems, the present invention discloses a target speech generation method and system.

[0006] The present invention discloses a target speech generation method, comprising the following steps: Obtaining a first voice, a second voice, and a target text; Extracting information features according to the first speech; extracting style features according to the second speech; Extracting phoneme corresponding features according to the target text; The target speech is generated according to the information features, style features and phoneme corresponding features.

[0007] Preferably, the second voice is a debate voice. Correspondingly, the target text is a rebuttal text that responds to the debate speech.

[0008] Preferably, the target speech is generated according to the information feature, style feature and phoneme correspondence feature, specifically: Acquire a speech data set and train a large language model based on the speech data set; Input the information features, style features, and phoneme corresponding features into a trained large language model, perform autoregressive prediction, and obtain target semantic features; A target speech is generated according to the target semantic features.

[0009] Preferably, generating the target speech according to the target semantic features is specifically as follows: A target speech is generated according to the target semantic features and a semantic-sound conversion algorithm.

[0010] Preferably, generating the target speech according to the target semantic features is specifically as follows: determining an acoustic token according to the target semantic feature; A target speech is generated according to the acoustic tokens.

[0011] Preferably, generating the target speech according to the acoustic token is specifically as follows: According to the acoustic tokens, a target speech is generated using a speech synthesis model or an audio synthesis technology.

[0012] Preferably, the target speech is generated according to the information feature, style feature and phoneme correspondence feature, specifically: Determining a debate style rule library according to the style characteristics, and selecting a preset debate style from the debate style rule library; The target speech is generated according to the preset debate style, the information features and the phoneme corresponding features.

[0013] Preferably, the target speech is generated according to the preset debate style, the information features and the phoneme corresponding features, specifically: Determining a speech modulation mode according to a preset debate style, the information characteristics, and the phoneme corresponding characteristics; According to the speech modulation mode, a target speech is generated.

[0014] When the second voice is a debate voice, the voice data set is a debate data set. Accordingly, a large language model is trained according to the voice data set, specifically: In the debate data set, the speech of the two debaters is separated to obtain the single audio of the two debaters; A large language model is trained based on the single-person audio of the two debaters.

[0015] The present invention also discloses a target speech generation system, including a speech and text acquisition module, a feature extraction module, a phoneme extraction module and a target speech generation module. The speech and text acquisition module is used to acquire the first speech, the second speech and the target text; The feature extraction module is used to extract information features according to the first speech and style features according to the second speech; The phoneme extraction module is used to extract phoneme corresponding features according to the target text; The target speech generation module is used to generate the target speech according to the information feature, style feature and phoneme corresponding feature.

[0016] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention integrates the style features of the second voice based on the information features of the first voice. Even when faced with limited training samples, the present invention can flexibly generate audio of any speaker in a variety of scenarios, which not only greatly enriches the dimension and depth of audio expression; but also this fusion of cross-voice features significantly improves the expressiveness and persuasiveness of the target voice, making it closer to the emotional concentration of real human communication, and can greatly increase the naturalness of the target voice, thereby resonating with the audience. (2) The present invention predicts the target semantic features by deeply fusing information features, style features and target text, thus achieving precise control of semantic content and acoustic details in the speech generation process, greatly improving the natural fluency of the generated speech; (3) The present invention can generate the target speech of any speaker in a preset scenario, so the present invention can easily control and adapt to a variety of speech styles; it shows excellent flexibility and adaptability when generating speech rich in specific emotions and styles; (4) The method and system of the present invention can provide real-time voice feedback for debate training, helping users improve their debate skills, thereby lowering the threshold for traditional debate training and enabling more users to improve their debate skills; (5) The target speech generated by the present invention is highly natural and expressive, so the present invention has the potential to be applied to any scenario that requires highly personalized speech generation, such as debates, movie dubbing, virtual assistants, interactive games, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a structural diagram of the system of the present invention; In the figure, 101 is a voice and text acquisition module; 102 is a feature extraction module; 103 is a phoneme extraction module; and 104 is a target voice generation module. DETAILED DESCRIPTION

[0018] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0019] like Figure 1 As shown, the present invention discloses a target speech generation method, comprising the following steps: S1, obtaining a first voice, a second voice and a target text, wherein the second voice at least includes the audio of one speaker; S2. Extracting information features according to the first speech, the extraction method is: in a speaker recognition system based on a Gaussian mixture model, first extracting the speaker's MFCC (Mel-Frequency Cepstral Coefficients) feature parameters, and then using the Gaussian mixture model to model and classify the feature parameters to obtain information features; in other embodiments, information features can also be extracted by using a Mel-frequency cepstral coefficient method, a linear prediction coefficient method, and a linear prediction cepstral coefficient method.

[0020] S3. Extracting style features according to the second speech. The style features can be extracted by using an Emotion-2Vec model, a Gaussian mixture model, a hidden Markov model, a convolutional neural network, or a recurrent neural network.

[0021] Preferably, in the present application, both information features and style features can be initially extracted by a semantic coder and then discretized by a quantization method.

[0022] S4, extracting phoneme corresponding features according to the target text, where the phoneme corresponding features can be extracted by short-time Fourier transform method, Mel frequency cepstrum coefficient method and Mel filter bank coefficient method; Preferably, the specific method for extracting phoneme corresponding features according to the target text may also be: adopting a text-to-phoneme method, such as Grapheme-to-Phoneme (G2P) and Byte Pair Encoding (BPE) algorithms to obtain a phoneme sequence, and then combining tools such as jieba and pypinyin of the Python programming language to obtain phoneme corresponding features according to the phoneme sequence.

[0023] S5. Generate target speech according to information features, style features and phoneme correspondence features.

[0024] The present invention integrates the style features of the second voice based on the information features of the first voice; even when faced with limited training samples, the present invention can flexibly generate audio of any speaker in a variety of scenarios, which not only greatly enriches the dimension and depth of audio expression; but also this fusion of cross-voice features significantly enhances the expressiveness and persuasiveness of the target voice, making it closer to the emotional concentration of real human communication, and can greatly increase the naturalness of the target voice, thereby resonating with the audience.

[0025] In the present invention, the first voice is the voice of any speaker; the second voice includes the voice of at least one speaker. The following is a description of the information features, style features and phoneme correspondence features: Information features include at least one of the speaker's personal characteristics (gender, age, identity); physical characteristics (frequency, amplitude, sound speed, wavelength); subjective perception characteristics (loudness, pitch, timbre); and prosodic characteristics (speech speed, sound quality, spectrum). The identities of different speakers can be identified through information features. Style features include at least one of the following: timbre style (sweet and gentle, bold and rough, steady and deep, lively and playful); sound quality characteristics (bright sound, dark sound, wide and narrow sound, thick and thin sound); emotion and expression style (emotional color, personalized expression); scene adaptability (speech speed, intonation, emotional characteristics, pauses and rhythm in different occasions); The phoneme correspondence features include the pronunciation position, pronunciation method, vocal cord vibration characteristics, and pronunciation duration of the initials and finals; The target text is the text corresponding to the target speech to be generated.

[0026] An example is as follows: the first voice is Xiaohong's speaking voice, and the second voice is Xiaoming's singing voice. Then the target voice is Xiaohong's singing voice in Xiaoming's style, and the target text is the lyrics of Xiaohong's singing.

[0027] The method of the present invention helps to generate purer and more universal speech by learning information features and style features respectively, and is particularly suitable for application scenarios that require strong style expression but do not require strong speaker dependence.

[0028] In Example 1, the second voice is a debate voice, and accordingly, the target text is a text that replies to the debate voice, and the target voice is a target debate voice that replies to the debate voice. Since the present invention focuses on learning the style of the second speaker using the voice of the first speaker, the target text can also be any text.

[0029] In Example 2, S5 is specifically: S51, obtaining a speech data set and training a large language model according to the speech data set; S52. Input the information features, style features and phoneme corresponding features into the trained large language model, perform autoregressive prediction, and obtain the target semantic features. In this embodiment, the target semantic features are obtained using an AR (auto-regressive) speech generation model based on the large language model framework.

[0030] In one embodiment, the speech data set includes single-person audio of two parties in the conversation (a first speaker and a second speaker), and the training process of the large language model is: extracting information features of the first speaker's audio, extracting style features of the second speaker's audio, extracting features corresponding to the first phoneme based on the speech text of the first speaker, and then inputting the information features, style features and features corresponding to the first phoneme into the large language model to obtain predicted semantic features, generating speech based on the predicted semantic features, using the first speaker's audio corresponding to the information features as the true value (ground truth), and then training the large language model.

[0031] When the dialogue scene is a debate, S51 is specifically: S511, separating the voices of the two parties of the debate according to the debate data set to obtain single-person audio of the two parties of the debate; S512. Train a large language model based on the single-player audio of the two debaters.

[0032] In the above embodiment, the single audio of the two parties in the conversation (or the two parties in the debate) is separately collected audio data, which is difficult to simulate the real scene and the state of the two parties. Therefore, in another embodiment, the process of establishing a voice data set is: collecting real-time audio of a real conversation or debate scene, using a speaker feature model such as speaker-diarization 3.1 to identify each speaker in the real-time audio, and then cutting the voice segments according to the recognition results to obtain the single audio corresponding to each speaker.

[0033] The real-time audio covers a wide range of emotions and tonal changes, ranging from calm narration to heated arguments, and each single-person audio is annotated in detail, including speech text information, the audio path of the debate context, and the audio duration, so as to provide the system of the present invention with an opportunity to learn natural speech changes in conversations and debates.

[0034] The speech data set of the present invention provides a valuable resource for speech synthesis research, and enables the method and system of the present invention to show good potential in processing multiple languages ​​and dialects.

[0035] S53: Generate target speech according to target semantic features.

[0036] In Example 3, S53 is specifically: The target speech is generated according to the target semantic features and the semantic-sound conversion algorithm. The semantic-sound conversion algorithm is any one of a vector quantization algorithm, a linear multivariate regression method, a SoundStorm model, a hidden Markov model or a Gaussian mixture model.

[0037] The present invention adopts a semantic-to-sound conversion algorithm to directly map the target semantic tag to a predefined acoustic feature library, which simplifies the semantic-to-sound conversion process and can provide faster processing speed and lower computing requirements in resource-constrained environments.

[0038] In Example 4, S53 is specifically: S531, determining an acoustic token according to a target semantic feature; S532: Generate target speech according to the acoustic token.

[0039] Preferably, S532 is specifically: According to the acoustic tokens, a speech synthesis model or audio synthesis technology is used to generate the target speech.

[0040] The speech synthesis model is any one of the following: SoundStorm model, VALL-E model, Tacotron model, and its extended model gst-tacotron; The audio synthesis technology is any one of deep neural network speech synthesis, text to speech (TTS), and statistical parametric speech synthesis (SPSS).

[0041] In other embodiments, the present invention also adopts serial or selective unit audio synthesis technology to generate target speech by splicing pre-recorded speech units to reduce costs and computing resource consumption.

[0042] The method of Example 5 is mainly applicable to the case where the second voice is a debate voice, and S5 is specifically: S51, determining a debate style rule library according to the style characteristics, and selecting a preset debate style from the debate style rule library; In Example 5, firstly, a debate data set is obtained, audio of a preset duration is selected from the debate data set as the second voice, and style features of the second voice are extracted; The debate style rule library is constructed as follows: according to the style characteristics of the second voice, a special feature classification model (such as the CNN-CTC model or the GRU-CTC model) is used to perform style classification, which includes at least one of the following: language style (gorgeous, plain), logical style (rigorous, loose), and attitude style (radical, gentle). After each classification, a manual description of the characteristics is made: including language expression, argumentation method, debater performance, etc.; the classified styles are stored in a database to form a debate style rule library; and signal processing technology or machine learning algorithms (such as clustering, classification, etc.) are used to identify speech modulation patterns of different debate styles from the second voice, and store them accordingly in the debate style rule library.

[0043] By inputting the characteristics corresponding to the desired debate style into the debate style rule library, a preset debate style can be obtained.

[0044] S52: Generate target speech according to the preset debate style, information features and phoneme correspondence features.

[0045] Preferably, S52 is specifically: S521, determining a speech modulation mode according to a preset debate style, information features, and phoneme corresponding features; S522: Generate target speech according to the speech modulation mode.

[0046] In this embodiment, the implementation process of S521 and S522 is as follows: first, a large language model or an NLP model is used to extract a speech modulation pattern corresponding to a preset debate style from a debate style rule library according to the preset debate style, information features, and phoneme corresponding features; then, corresponding speech synthesis model parameters are selected or adjusted according to the speech modulation pattern; finally, the speech synthesis model in Example 4 is used to generate a speech output with a specific debate style.

[0047] The present invention utilizes a debate style rule library to simulate the style and emotion of debate speech, and through rule matching and application of predefined speech modulation patterns, style adaptation can be quickly achieved without large-scale data support.

[0048] like Figure 2 As shown, the present invention also discloses a target speech generation system, including a speech and text acquisition module 101, a feature extraction module 102, a phoneme extraction module 103 and a target speech generation module 104; The speech and text acquisition module 101 is used to acquire the first speech, the second speech and the target text; The feature extraction module 102 is used to extract information features according to the first speech and style features according to the second speech; The phoneme extraction module 103 is used to extract phoneme corresponding features according to the target text; The target speech generation module 104 is used to generate the target speech according to the information feature, the style feature and the phoneme correspondence feature.

[0049] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention integrates the style features of the second voice based on the information features of the first voice. Even when faced with limited training samples, the present invention can flexibly generate audio of any speaker in a variety of scenarios, which not only greatly enriches the dimension and depth of audio expression; but also this fusion of cross-voice features significantly improves the expressiveness and persuasiveness of the target voice, making it closer to the emotional concentration of real human communication, and can greatly increase the naturalness of the target voice, thereby resonating with the audience. (2) The present invention predicts the target semantic features by deeply fusing information features, style features and target text, thus achieving precise control of semantic content and acoustic details in the speech generation process, greatly improving the natural fluency of the generated speech; (3) The present invention can generate the target speech of any speaker in a preset scenario, so the present invention can easily control and adapt to a variety of speech styles; it shows excellent flexibility and adaptability when generating speech rich in specific emotions and styles; (4) The method and system of the present invention can provide real-time voice feedback for debate training, helping users improve their debate skills, thereby lowering the threshold for traditional debate training and enabling more users to improve their debate skills; (5) The target speech generated by the present invention is highly natural and expressive, so the present invention has the potential to be applied to any scenario that requires highly personalized speech generation, such as debates, movie dubbing, virtual assistants, interactive games, etc.

[0050] The above are only a few embodiments of the present application and do not constitute any form of limitation to the present application. Although the present application is disclosed as above with preferred embodiments, it is not intended to limit the present application. Any technician familiar with the profession, without departing from the scope of the technical solution of the present application, using the technical content disclosed above to make slight changes or modifications are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A target speech generation method, characterized in that: The following steps are involved: Obtaining a first voice, a second voice, and a target text; Extracting information features according to the first speech; extracting style features according to the second speech; Extracting phoneme corresponding features according to the target text; The target speech is generated according to the information features, style features and phoneme corresponding features.

2. The target speech generation method according to claim 1, characterized in that: The second voice is a debate voice, Correspondingly, the target text is a text that responds to the debate voice.

3. The target speech generation method according to claim 1 or 2, characterized in that: Generate the target speech according to the information features, style features and phoneme corresponding features, specifically: Acquire a speech data set and train a large language model based on the speech data set; Input the information features, style features, and phoneme corresponding features into a trained large language model, perform autoregressive prediction, and obtain target semantic features; A target speech is generated according to the target semantic features.

4. The target speech generation method according to claim 3, characterized in that: Generating a target speech according to the target semantic feature is specifically as follows: A target speech is generated according to the target semantic features and a semantic-sound conversion algorithm.

5. The target speech generation method according to claim 3, characterized in that: Generating a target speech according to the target semantic feature is specifically as follows: determining an acoustic token according to the target semantic feature; A target speech is generated according to the acoustic tokens.

6. The target speech generation method according to claim 5, characterized in that: Generate the target speech according to the acoustic token, specifically: According to the acoustic tokens, a target speech is generated using a speech synthesis model or an audio synthesis technology.

7. The target speech generation method according to claim 2, characterized in that: Generate the target speech according to the information features, style features and phoneme corresponding features, specifically: Determining a debate style rule library according to the style characteristics, and selecting a preset debate style from the debate style rule library; The target speech is generated according to the preset debate style, the information features and the phoneme corresponding features.

8. The target speech generation method according to claim 7, characterized in that: Generate target speech according to the preset debate style, the information features and the phoneme corresponding features, specifically: Determining a speech modulation mode according to a preset debate style, the information characteristics, and the phoneme corresponding characteristics; According to the speech modulation mode, a target speech is generated.

9. The target speech generation method according to claim 3, characterized in that: When the second voice is a debate voice, the voice data set is a debate data set. Accordingly, a large language model is trained according to the voice data set, specifically: In the debate data set, the speech of the two debaters is separated to obtain the single audio of the two debaters; A large language model is trained based on the single-person audio of the two debaters.

10. A target speech generation system, characterized in that: It includes speech and text acquisition module, feature extraction module, phoneme extraction module and target speech generation module. The speech and text acquisition module is used to acquire the first speech, the second speech and the target text; The feature extraction module is used to extract information features according to the first speech and style features according to the second speech; The phoneme extraction module is used to extract phoneme corresponding features according to the target text; The target speech generation module is used to generate the target speech according to the information feature, style feature and phoneme corresponding feature.

Images