Model training method and device, voice generation method and device, equipment and medium

Through model training and consistency knowledge distillation technology, voice with the sound characteristics of the target object is generated, which solves the problem of single speech expression of virtual objects in the prior art, improves the efficiency and authenticity of speech generation, and enhances the user experience.

CN118942447APending Publication Date: 2024-11-12BEIJING PERFECT WORLD SOFTWARE TECH DEV CO LTD
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202410814367.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-21
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the existing voice interaction technology, the voice expression of virtual objects is too single and cannot interact through voice like real people, making it difficult for users to obtain a real interactive experience.

Method used

Through the model training method, the phoneme feature representation of the target text and the sound embedding representation of the target object are generated by the prior phoneme feature distribution parameters carrying the sound characteristics of the target object, and consistency knowledge distillation training is carried out to obtain the sound characteristics of the fused target object that can be generated speech model.

Benefits of technology

It improves the efficiency and authenticity of speech generation, makes the speech expression of virtual objects more natural and diversified, and enhances the user's interactive experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942447A_ABST
    Figure CN118942447A_ABST
Patent Text Reader

Abstract

The invention discloses a model training method and device, a voice generation method and device, equipment and a medium. The method comprises the following steps: generating prior phoneme feature distribution parameters carrying target object sound characteristics according to phoneme feature representation corresponding to a target text and embedded representation of target object sound; inputting the embedded representation of the sound of the target object and the priori phoneme feature distribution parameters into the first model for consistent knowledge distillation training to obtain a second model; wherein the second model learns the ability of generating the fused target object sound characteristic prediction voice from the first model. In the training process, the target object is embedded and input into different models, so that the natural language expression ability of the voice information output by the models is integrally improved, and the voice information can more accurately reflect the sound characteristics of the target object. In addition, the second model is obtained through training in a knowledge distillation mode, so that the training efficiency and the voice generation efficiency in the subsequent model application process can be effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of Internet technology, and in particular to model training, speech generation methods, devices, equipment and media. Background Art

[0002] In human-computer interaction scenarios, users can interact with virtual objects in a variety of ways. In order to give users a better interactive experience, virtual objects can respond through actions, expressions, language, and other methods.

[0003] Among them, the interaction between virtual objects and users through voice is a common interaction method. However, in the existing voice interaction process, the voice expression of virtual objects is often relatively simple. For example, the voices expressed by different virtual objects are relatively similar, or virtual characters of different genders are represented only by different timbres. It is difficult for users to obtain a real interactive experience without being able to interact through voice like real people. Therefore, a solution is needed to quickly generate more realistic voice information. Summary of the invention

[0004] The embodiments of the present invention provide a model training, speech generation method, device, equipment and medium to improve the efficiency and real effect of generating speech information.

[0005] In a first aspect, an embodiment of the present invention provides a model training method, the method comprising:

[0006] Generate a priori phoneme feature distribution parameters carrying the sound characteristics of the target object according to the phoneme feature representation corresponding to the target text and the embedded representation of the target object sound;

[0007] The embedded representation of the target object sound and the prior phoneme feature distribution parameters are input into the first model for consistency knowledge distillation training to obtain a second model; wherein the second model learns from the first model the ability to generate speech predicted by integrating the target object sound characteristics.

[0008] Optionally, the step of inputting the embedded representation of the target object sound and the prior phoneme feature distribution parameters into the first model for consistent knowledge distillation training to obtain the second model comprises:

[0009] Inputting the priori phoneme feature distribution parameters into the first model for forward diffusion processing to obtain noise distribution parameters including noise information;

[0010] Sampling the noise distribution parameters to obtain a sampling trajectory;

[0011] Performing reverse denoising processing on the sampling trajectory using the first solver and the embedded representation of the target object sound to obtain a predicted sample incorporating the target object sound characteristics;

[0012] When it is determined that the predicted sample meets the requirements, consistent knowledge distillation training is performed to obtain the second model.

[0013] Optionally, when it is determined that the predicted sample meets the requirement, performing consistency knowledge distillation training to obtain the second model includes:

[0014] When the first loss function is used to determine that the predicted sample meets the requirement, a first model that has been trained is obtained;

[0015] The first model is used to perform consistency knowledge distillation training on the model to be trained, so as to obtain the second model that learns from the first model the ability to generate speech prediction by integrating the sound characteristics of the target object starting from any point in the sampling trajectory.

[0016] Optionally, generating a priori phoneme feature distribution parameters carrying the target object sound characteristics according to the phoneme feature representation corresponding to the target text and the embedded representation of the target object sound includes:

[0017] Converting the phoneme feature representation into a first phoneme vector;

[0018] Concatenating the first phoneme vector with the embedded representation of the target object sound to obtain a second phoneme vector;

[0019] Inputting the second phoneme vector into an encoder to generate an encoded phoneme vector;

[0020] The encoded phoneme vector is processed to obtain the prior phoneme feature distribution parameter carrying the predicted duration.

[0021] Optionally, the processing of the encoded phoneme vector to obtain the prior phoneme feature distribution parameter carrying the predicted duration includes:

[0022] Processing the encoded phoneme vector to obtain a distribution parameter to be corrected;

[0023] The distribution parameters to be corrected are corrected using a second loss function to obtain the prior phoneme feature distribution parameters carrying the predicted duration.

[0024] Optionally, the using of a second loss function to correct the distribution parameter to be corrected to obtain the priori phoneme feature distribution parameter carrying the predicted duration includes:

[0025] Processing the distribution parameters to be corrected to obtain an alignment matrix;

[0026] Converting the alignment matrix to unsupervised duration;

[0027] Determining a predicted duration according to the encoded phoneme vector;

[0028] Input the unsupervised duration and the predicted duration into the second loss function to calculate the corresponding second loss value;

[0029] When the second loss value does not meet the second loss threshold, correcting the to-be-corrected distribution parameter based on the alignment matrix;

[0030] When the second loss value meets the second loss threshold, the correction is terminated to obtain the prior phoneme feature distribution parameter carrying the predicted duration.

[0031] In a second aspect, an embodiment of the present invention provides a speech generation method, the method comprising:

[0032] Obtaining embedded representations of the text information to be processed and the target object's sound;

[0033] Converting the text information into a target phoneme feature representation;

[0034] Generate a priori phoneme feature distribution parameters carrying the target object sound characteristics according to the target phoneme feature representation and the embedded representation of the target object sound;

[0035] The target object index corresponding to the prior phoneme feature distribution parameter and the embedded representation of the target object sound is input into a trained second model, and speech information fused with the target object sound characteristics is output.

[0036] Optionally, generating a priori phoneme feature distribution parameters carrying the sound characteristics of the target object according to the target phoneme feature representation and the target object embedding includes:

[0037] Inputting the target phoneme feature representation fused with the embedded representation of the target object sound into an encoder to generate a third phoneme vector;

[0038] Determine a predicted phoneme duration according to the third phoneme vector;

[0039] Inputting the third phoneme vector into the convolution layer to obtain a priori phoneme feature distribution parameter to be corrected;

[0040] The predicted phoneme duration is used to correct the prior phoneme feature distribution parameters to be corrected, so as to generate prior phoneme feature distribution parameters carrying the sound characteristics of the target object.

[0041] Optionally, the step of inputting the target object index corresponding to the prior phoneme feature distribution parameter and the embedded representation of the target object sound into a trained second model, and outputting speech information fused with the target object sound characteristics, comprises:

[0042] Inputting the prior phoneme feature distribution parameter into the second model for forward diffusion processing to obtain a target noise distribution parameter containing noise information;

[0043] The target noise distribution parameters are subjected to reverse denoising processing using the target object index corresponding to the embedded representation of the target object sound, so as to obtain speech information integrated with the target object sound characteristics.

[0044] In a third aspect, an embodiment of the present invention provides a model training device, the device comprising:

[0045] A generation module, used to generate a priori phoneme feature distribution parameters carrying the sound characteristics of the target object according to the phoneme feature representation corresponding to the target text and the embedded representation of the target object sound;

[0046] A training module is used to input the embedded representation of the target object sound and the prior phoneme feature distribution parameters into the first model for consistency knowledge distillation training to obtain a second model; wherein the second model learns from the first model the ability to generate speech prediction that integrates the target object sound characteristics.

[0047] In a fourth aspect, an embodiment of the present invention provides a speech generating device, the device comprising:

[0048] An acquisition module, used to acquire the text information to be processed and the embedded representation of the target object sound;

[0049] A conversion module, used for converting the text information into a target phoneme feature representation;

[0050] A generating module, configured to generate a priori phoneme feature distribution parameters carrying the target object sound characteristics according to the target phoneme feature representation and the embedded representation of the target object sound;

[0051] The output module is used to input the target object index corresponding to the prior phoneme feature distribution parameter and the embedded representation of the target object sound into the trained second model, and output the speech information integrated with the target object sound characteristics.

[0052] In a fifth aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the model training method described in the first aspect or the speech generation method described in the second aspect.

[0053] In the sixth aspect, an embodiment of the present invention also provides a computer-readable medium on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the model training method described in the first aspect or the speech generation method described in the second aspect.

[0054] In an embodiment of the present invention, in order to enable the trained model to have a higher speech generation efficiency and a more realistic speech generation effect, during the training process, the embedded representation of the target object sound is input to the relevant model in each link. Specifically, the pre-prepared target text is processed into a phoneme feature representation, and then the phoneme feature representation and the embedded representation of the target object sound used for training are input to the encoder to be trained, and the encoder outputs the phoneme feature distribution parameters that incorporate the target object sound characteristics. Further, the phoneme feature distribution parameters and the embedded representation of the target object sound are input to the first model for knowledge distillation training to obtain a second model. During the training process, the embedded representation of the target object sound is input into different models, thereby achieving an overall improvement in the natural language expression ability of the speech information output by the model, so that the language information can more accurately reflect the target object sound characteristics. In addition, the second model is trained by knowledge distillation, which can effectively improve the training efficiency and the speech generation efficiency in the subsequent model application process. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0057] Figure 2 A second model training method provided in an embodiment of the present application;

[0058] Figure 3 A schematic diagram of a knowledge distillation training framework provided in an embodiment of the present application;

[0059] Figure 4 A schematic diagram of a process for generating phoneme feature distribution parameters provided in an embodiment of the present application;

[0060] Figure 5 A flowchart of a speech generation method provided in an embodiment of the present application;

[0061] Figure 6 A schematic diagram of the flow of speech generation for illustrating an embodiment of the present application;

[0062] Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0063] Figure 8 A schematic diagram of the structure of a speech generating device provided in an embodiment of the present application;

[0064] Fig. 9 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0067] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0068] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0069] In a virtual interactive scene, users can watch the various actions of virtual targets in the scene, and some users can also interact with virtual objects to gain a greater sense of participation. For example, in a game scene, players (that is, users) need to receive a large amount of game guide information, navigation information, etc., so that game players can get a more intuitive guidance effect. In addition, there are non-player characters (NPC) in the game, in which the plot and dialogue are important parts of the game. NPC characters will use text to speech (TTS) technology to interact with game players, so that players can get a better gaming experience.

[0070] For example, in the human-computer interaction scenario in a bank lobby, users can directly interact with the robot through voice when handling business. The robot uses TTS technology to synthesize voice during voice interaction, which not only helps users successfully handle business, but also allows users to have a better interactive experience.

[0071] In addition to the scenarios listed above, there are many scenarios where virtual characters need to communicate with users through voice, which is more efficient and direct than text communication. However, the existing TTS technology is relatively simple, and the synthesized voice has obvious robot traces (single speech speed, intonation, timbre, etc.), and it is difficult to distinguish different robots through synthesized voice. For users, the interactive experience is monotonous and stereotyped. Therefore, a solution for quickly generating voice information is needed.

[0072] To facilitate understanding, specific embodiments will be used as examples below.

[0073] like Figure 1 A flow chart of the model training method provided in the embodiment of the present application. Figure 1 As you can see, the specific steps include the following:

[0074] Step 101: Generate prior phoneme feature distribution parameters carrying target object sound characteristics based on the phoneme feature representation corresponding to the target text and the embedded representation of the target object sound.

[0075] The target object sound characteristics may be information representing the sound characteristics obtained based on a sound-emitting body such as a person, an animal, or an object that can emit sound. It is understandable that the target object may be a virtual object or a physical object, and the virtual object may be a virtual character, for example, the target object may be an NPC in a game, an intelligent customer service, an intelligent voice assistant, a virtual character in an intelligent teaching aid, etc. Different target objects are given different sound characteristics.

[0076] Step 102: Input the embedded representation of the target object's voice and the prior phoneme feature distribution parameters into the first model for knowledge distillation training to obtain a second model; wherein, the second model learns from the first model the ability to generate predicted speech that fuses the characteristics of the target object's voice.

[0077] The target text mentioned here can be text content prepared in advance by staff. For example, in a game scenario, target texts prepared in advance for NPCs according to different storylines. In addition, the target text can also be Q&A text generated during the interaction with the user. For example, if the user's question is "What time is it now?", the target text of the corresponding virtual character can be "It is 2 pm now". In addition, this solution can also be used in other scenarios. The content of the target text prepared for different scenarios may be different, and the language of the target text may also be different.

[0078] After obtaining the target text, further convert the target text into a phoneme feature representation. Embodiments of the present invention can be applied to various language systems (such as various languages like Chinese, English, Japanese, etc.). Different language systems result in different forms of phoneme feature representations. As an optional solution, assuming the target text is in Chinese, convert the target text into a tone-marked pinyin sequence through the pypinyin library (pinyin library) of Python. Then, convert the tone-marked pinyin sequence into the corresponding phoneme sequence through the Chinese phoneme table; perform short-time Fourier transform (STFT) on the phoneme sequence to extract the corresponding Mel spectrogram. Specifically,

[0079] Using the pypinyin library of Python to convert the input target text into a tone-marked pinyin representation and converting the pinyin into a phoneme sequence through the Chinese phoneme table is a process of converting text into a phoneme representation.

[0080] Pypinyin, as a Python library, can convert Chinese character strings into pinyin and supports tone marking, which helps to retain the speech information in the original text. For example, if the target text is "Hello, world", the standard tone-marked pinyin output can be [['nǐ'], ['hǎo'], [','], ['shì'], ['jiè']], where each Chinese character is converted into a tone-marked pinyin.

[0081] In order to convert pinyin with tones into phoneme sequences, a mapping table (i.e., Chinese phoneme table) can be established, which maps each pinyin (including tones) to a corresponding phoneme sequence. This mapping table needs to be constructed based on the knowledge of Chinese phonetics, because different pinyin and tones may correspond to different phoneme combinations. In addition, in order to meet the diverse usage needs of users, when establishing a mapping table, a corresponding mapping table can be established based on the distribution of dialect characteristics in different regions.

[0082] For example, the mapping table of the standard language: 'nǐ':['ni3'](ni3 represents the phonetic representation of 'you'), 'hǎo':['hao3'](hao3 represents the phonetic representation of 'good').

[0083] As an alternative, assuming that the target text is in English, the phoneme sequence corresponding to each word is found through a dictionary (Lexicon). For example, the English word "hello" may be mapped to the phoneme sequence / hh AH0l OW1 / (note that this is just an example, and the actual phoneme representation may vary from system to system). In practical applications, ready-made tool libraries are usually used to help achieve text-to-phoneme conversion. For example, you can use nltk.corpus.cmudict in the NLTK (Natural Language Toolkit) library to access the CMU Pronouncing Dictionary, which contains a large number of English words and their phoneme mappings. Using these tool libraries can greatly simplify the text-to-phoneme conversion process and improve the accuracy and efficiency of the conversion.

[0084] After obtaining the phoneme sequence (which can be understood as the phoneme sequence obtained from various languages) using the above method, it is further processed using short-time Fourier transform, and the Fourier transform of each frame is calculated to obtain the phoneme feature representation of the time-frequency signal, such as the Mel spectrum. The Mel spectrum (phoneme feature representation) is a feature representation obtained by transforming the audio signal based on the auditory characteristics of the human ear. In addition, there are many other phoneme feature representation methods, such as Linear Predictive Coding (LPC), Spectral Centroid, etc.

[0085] The target object's voice embedding (Speaker Embedding, spk_emb) is used to represent the voice characteristics of the target object (that is, the speaker). This embedding is usually a fixed-dimensional vector extracted from the target object's voice sample by training the model. This vector can capture the individual characteristics of the target object, allowing the model to take the target object's information into account when processing the voice.

[0086] As an optional solution, the previously obtained phoneme feature representation and the embedded representation of the target object sound can be input into the encoder, the encoder is trained, and the prior phoneme feature distribution parameters are input. The encoder mentioned here can be an unsupervised learning encoder or a supervised learning encoder. As a type of unsupervised learning encoder, the autoencoder can share weights, that is, the weights of the decoder and the encoder are transposed to each other, which can speed up the network learning because the number of training parameters is reduced. The encoder usually includes an input layer, a hidden layer (encoding layer) and an output layer (decoding layer). The input layer receives the original data (that is, the phoneme feature representation generated after text preprocessing), the encoding layer compresses the data into a low-dimensional feature representation, and the decoding layer attempts to reconstruct the original data from the compressed features. In the encoder structure, some special components may also be included, such as regularization terms, noise layers, etc., which are used to enhance the generalization ability of the model or prevent overfitting.

[0087] When training the encoder, the phoneme feature representation (for example, the Mel spectrum mentioned above) and the embedded representation of the target object sound are input into the encoder to be trained, so that the encoder can output the prior phoneme feature distribution parameters (that is, Mel spectrum distribution parameters μ) that are integrated with the sound characteristics of the target object. In order to make the Mel spectrum distribution parameters μ better integrate the sound characteristics of the target object, it is necessary to use the duration prediction component to correct the Mel spectrum distribution parameters μ. It should be noted that the prior phoneme feature distribution parameters can be time domain distribution feature parameters, frequency domain distribution feature parameters or time-frequency distribution features, such as Mel spectrum distribution parameters μ.

[0088] After obtaining the available prior phoneme feature distribution parameters in the above manner, the second model needs to be trained next. The first model mentioned here is the teacher model, and the corresponding second model is the student model. For example, the first model and the second model can be diffusion models (for example, distillation diffusion models), or Poisson flow generation models, etc.

[0089] During training, the embedded representation of the target object's voice and the prior phoneme feature distribution parameters are simultaneously input into the first model to be trained, so that when the trained first model is used to generate speech, the generated speech information contains the target object's voice characteristics, making the synthesized speech more natural and more like the speech expressed by a real person (for example, with different tones, speech speed, timbre, etc.). By using knowledge distillation (or consistency distillation), the first model and the second model satisfy the following equation at any time t and t':

[0090] D(x t ,t)=D(x t′ ,t′)

[0091] D(x0,0)=x0

[0092] Among them, the parameter x t represents the noise distribution state at time t, parameter x t′ represents the noise distribution state at time t', parameter x0 represents the noise distribution state at the initial time, D(x t ,t) represents the denoising result of x_t at time t, and D(x0,0) represents the denoising result of x_0 at the initial time. The second model obtained through knowledge distillation training has the same speech processing capability as the first model, that is, the speech output by the second model contains the target object sound characteristics. The target object sound characteristics mentioned here can be understood as the timbre, pitch, phoneme duration (speech pauses), speech speed, etc. of the target object, which can significantly distinguish the speech of different people.

[0093] Through the above scheme, during training, the embedded representation of the target object sound is first input into the encoder to be trained, so that the prior phoneme feature distribution parameters output by the encoder carry the target object sound characteristics. Further, when training the second model, the target object embedding and the prior phoneme feature distribution parameters are also used as input, so that the trained second model has the ability to output speech information that integrates the target object sound characteristics. In addition, since the second model is a student model, while learning from the first model the ability to generate speech information containing the target object sound characteristics, the second model is smaller than the first model, and the second model can retain the performance of the first model while reducing computing resource consumption.

[0094] like Figure 2 A second model training method is provided in an embodiment of the present application. Figure 2 As shown, the embedded representation of the target object sound and the prior phoneme feature distribution parameters are input into the first model for knowledge distillation training to obtain the second model, which specifically includes the following steps:

[0095] Step 201: Input the prior phoneme feature distribution parameters into the first model for forward diffusion processing to obtain noise distribution parameters including noise information.

[0096] Step 202: Sampling the noise distribution parameters to obtain a sampling trajectory.

[0097] Step 203: Perform reverse denoising on the sampling trajectory using the first solver and the embedded representation of the target object sound to obtain a predicted sample that incorporates the target object sound characteristics.

[0098] Step 204: When it is determined that the predicted sample meets the requirement, consistent knowledge distillation training is performed to obtain the second model.

[0099] As described in step 204 , when it is determined that the predicted sample meets the requirements, consistent knowledge distillation training is performed to obtain the second model, which specifically includes: step 2041 and / or step 2042 .

[0100] Step 2041: When the first loss function is used to determine whether the predicted sample meets the requirements, a first model that has been trained is obtained.

[0101] Step 2042: Use the first model to perform consistency knowledge distillation training on the model to be trained, and obtain the second model that learns from the first model the ability to generate and predict speech from any point in the sampling trajectory by integrating the sound characteristics of the target object.

[0102] For ease of understanding, the following embodiments will be described by taking the first model as a diffusion model as an example.

[0103] The diffusion model training process can be divided into two processes: forward diffusion and reverse denoising. In the forward diffusion process, the noise is gradually added to the sampled data in the source distribution to make it a noise sample in a specific noise distribution (usually a Gaussian distribution). In the reverse denoising process, the noise data is randomly sampled from the noise distribution, and the noise is gradually denoised to finally obtain the sample data in the target distribution.

[0104] For example, after forward diffusion (i.e., parameterization) of the prior phoneme feature distribution parameter (i.e., the prior Mel spectrum distribution parameter μ), the noise distribution Xn (where X n ~N(μ,I), which is a Gaussian distribution). Where Xn obeys a Gaussian distribution with mean μ and covariance matrix I (unit matrix), that is, X_n~N(μ,I). Here, μ is the mean or other feature of the Mel spectrum, and I indicates that the covariance matrix of the noise is the unit matrix, which means that the noise in each dimension is independent and has the same variance. In practical applications, the above forward diffusion process can be expressed as an analytical form of a stochastic differential equation (SDE):

[0105] dx=f(x,t)dt+g(t)dw

[0106] where f(·) and g(·) are the drift coefficient and diffusion coefficient, respectively, w is the random Wiener process, and t is the time.

[0107] Furthermore, the noise distribution parameters are sampled. The stochastic differential equation SDE corresponds to a probability flow ordinary differential equation (ODE) that represents the sampling trajectory distribution of the SDE at time t:

[0108]

[0109] Where Pt(x) is the marginal probability density of x, For p t (x), the above probability flow ordinary differential equation (ODE) eliminates the random process w, thereby generating a deterministic sampling trajectory dx.

[0110] As mentioned above, the sampling trajectory contains noise and needs further denoising. In the denoising process, the sampling trajectory is subjected to reverse denoising using the first solver (i.e., the probability flow ordinary differential equation (ODE)) and the embedded representation of the target object sound to obtain a predicted sample that incorporates the target object sound characteristics. Specifically,

[0111] Here we define D(x t ,t) is x t The denoising function at time t, then the score function can be obtained by minimizing the denoising error

[0112]

[0113] in, For p t (x) is the fractional function, x t represents the noise distribution state at time t, represents the variance of the noise distribution at time t.

[0114] Furthermore, the sampling process based on the probability flow ordinary differential equation (ODE) can firstly sample from the noise distribution parameters and then perform conditional denoising prediction through the first solver combined with the embedded representation spk_emb with the target object sound to obtain the predicted sample. At this time, D(xt,t) becomes D(xt,t) t ,t,cond). The cond in D(x_t,t,cond) represents the embedded representation of the target object sound. The purpose of the denoising function D(xt,t) is to predict the denoised data based on the noisy data xt at the current time t and the conditional information cond (i.e., the embedded representation of the target object sound). Therefore, D(xt,t,cond) represents a conditional denoising function, which uses the embedded representation of the target object sound (information representing the characteristics of the target object) to guide the denoising process, thereby generating speech with a specific target object style. In the sampling process of the diffusion model, the first solver and the embedded representation of the target object sound are used as conditional information to gradually restore samples with specific target object characteristics from the noise.

[0115] In the ODE-based sampling process, the first solver plays the role of simulating the denoising process. It updates the initial noise samples sampled from the noise distribution in an iterative manner according to the predefined ODE equation, so that it gradually approaches the target distribution. In this process, the embedded representation of the target object sound can be introduced as conditional information to affect the ODE solution process.

[0116] Specifically, the target object information is usually embedded by using the target object embedding vector as an input feature of the model. In each step of the ODE solution process, the model will take this input feature into account and adjust its denoising strategy to ensure that the generated samples retain the characteristics of the target object while removing noise. In this way, by combining the numerical ODE solver and the target object embedding, conditional speech sample generation can be achieved.

[0117] In practice, embedding the target object information may require some specific techniques or methods. For example, deep learning models (such as recurrent neural networks or convolutional neural networks) can be used to extract the target object embedding and combine it with other features or parameters in the ODE solver. The specific embedding method and implementation details may vary depending on different models and tasks.

[0118] In summary, using numerical ODE solvers and target object embedding as conditional information is a method of guiding sample generation by combining these two elements during the sampling process. In this way, we can generate speech samples that better meet specific conditions and improve the effect and performance of tasks such as speech synthesis or speech recognition.

[0119] The predicted sample represents the speech information corresponding to the predicted target text. The loss function can be used to determine whether the predicted sample meets the requirements. If not, the above steps need to be repeated for iterative training. If it meets the requirements, the training is stopped.

[0120] After obtaining the prediction samples, the second model training can be performed. Figure 3 Schematic diagram of the knowledge distillation training framework provided in the embodiment of the present application. Figure 3It can be seen that it includes the first model (teacher model) and the second model (student model) to be trained. The prior phoneme feature distribution parameters X_n~N(μ,I) are input into the first model, and Noise_mel is obtained through forward diffusion processing. Then, the second model is trained by the consistency distillation method while being processed by the reverse denoising method, and the predicted sample Pred_mel is output. GT_mel is the reference sample for calculating the loss function value loss. The first loss function is used to determine whether the predicted sample meets the requirements. The first loss function is used to enable the second model to better sample and denoise the predicted sample from the noisy prior Mel spectrum distribution parameters (that is, the noise distribution parameters), that is, the real Mel spectrum, so as to obtain a more diverse, vivid and natural speech generation effect.

[0121] In one or more embodiments of the present application, the step of inputting the phoneme feature representation and the embedded representation of the target object sound into an encoder to generate a priori phoneme feature distribution parameters that incorporate the target object sound characteristics includes:

[0122] Converting the phoneme feature representation into a first phoneme vector;

[0123] Concatenating the first phoneme vector with the embedded representation of the target object sound to obtain a second phoneme vector;

[0124] Inputting the second phoneme vector into an encoder to generate an encoded phoneme vector;

[0125] The encoded phoneme vector is input into the convolution layer and the duration prediction component to generate the prior phoneme feature distribution parameters integrated with the sound characteristics of the target object.

[0126] like Figure 4 A schematic diagram of a process for generating a priori phoneme feature distribution parameters provided in an embodiment of the present application. Figure 4 As can be seen in Figure 1, the vector conversion module Phoneme embedding is used to vectorize the provided phoneme feature representation to obtain a phoneme vector. Then, the phoneme vector is input into the language synthesis module to obtain the first phoneme vector.

[0127] After obtaining the first phoneme vector, the first phoneme vector and the previously prepared embedded representation of the target object sound are spliced. There are many ways to splice the first phoneme vector with the embedded representation of the target object sound (also a representation in vector form), for example, direct splicing or splicing using a specific programming language. Direct splicing: The first phoneme vector and the embedded representation vector of the target object sound can be directly spliced ​​together in a certain order. For example, the splicing of the first phoneme vector u=[u1,u2,...,um] and the embedded representation vector v=[v1,v2,...,vn] of the target object sound can be expressed as u∥v=[u1,u2,...,um,v1,v2,...,vn]. It should be noted that when splicing, compatibility in certain dimensions needs to be met. By splicing vectors from different sources or different representations together, a richer and more comprehensive feature representation can be created, thereby improving the performance of the model. In the embodiment of the present application, the first phoneme vector is concatenated with the embedded representation of the target object sound in order to enhance the ability of the encoder to output Mel-spectrogram parameters that carry the target object sound characteristics.

[0128] In the encoder, there are multiple layers, such as Multi-Head Attention (MHA), LayerNorm, Feed-Forward Network (FFN), etc. Among them, Multi-Head Attention (MHA) is an attention mechanism that enhances the model's attention to different features by extending attention to multiple "heads". Each head can independently focus on different parts of the input sequence to capture richer information. MHA can help the model better understand and process sequence data.

[0129] Layer Normalization LayerNorm is a regularization technique commonly used in deep learning models. It improves the performance and stability of the model by normalizing each layer in the neural network. LayerNorm normalizes the input of each neuron so that the input of the neural network has a fixed distribution. This helps to accelerate the convergence of the neural network and improve the gradient flow.

[0130] Feed-Forward Network (FFN) In many deep learning models, FFN usually consists of two fully connected layers with ReLU activation function. The main function of FFN is to "expand" and "compress" the input representation. The first fully connected layer maps the input to a higher dimension, while the second fully connected layer maps this higher dimensional representation back to the original dimension.

[0131] After being processed by the above encoder, an encoded phoneme vector is generated. The encoded phoneme vector is input to the convolution layer (Conv_layer) and the duration prediction component (Duration Predictor) respectively. The specific implementation method includes: inputting the encoded phoneme vector to the convolution layer to obtain the distribution parameter to be corrected; inputting the encoded phoneme vector to the duration prediction component, correcting the distribution parameter to be corrected using the second loss function, and obtaining the corrected prior phoneme feature distribution parameter carrying the sound characteristics of the target object.

[0132] In this embodiment, the encoded phoneme vector is input into the duration prediction component (DurationPredictor) and then the predicted phoneme duration (Duration) is output. The Duration Predictor here plays a vital role in predicting the duration of phonemes. The following are its main functions: The main function of Duration Predictor is to predict the duration of each phoneme, that is, the repetition multiple in the later stage, and provide alignment information for the length regulator. This is crucial to ensure the accurate alignment between phonemes and speech waveforms in speech synthesis, which helps to improve the naturalness and expressiveness of speech synthesis.

[0133] In the inference phase, the duration sequence generated by the Duration Predictor can be used as the input of the length regulator to generate the MEL spectrum. In addition, by adjusting the multiplier of the prediction result of the duration predictor, the synthesis speed can be controlled, thereby optimizing the speech synthesis speed.

[0134] The structural design of Duration Predictor, such as the use of multi-layer FFT and linear layers, and the use of statistical language model training methods to jointly train sound vectors and word vectors, helps to more accurately describe text features and retain the coherence characteristics of the target text corresponding to the speech when simulating the sound characteristics of the target object. These technologies have greatly improved the accuracy of duration prediction, thereby improving the overall effect of speech synthesis. Duration Predictor plays a key role in speech synthesis. By predicting the duration of phonemes, it provides accurate alignment information and duration adjustment capabilities for speech synthesis, which helps to improve the naturalness and expressiveness of speech synthesis, that is, to make the synthesized speech more like the speaking style and tone of the specified target object.

[0135] After the encoded phoneme vectors are input to the convolution layer (Conv_layer), the output is the distribution parameters to be corrected. The distribution parameters to be corrected here cannot accurately express the duration corresponding to each phoneme. In other words, if the correction parameters are not corrected by predicting the duration, the actual duration of each phoneme will be incorrect, affecting the accuracy of the target object sound contained in the subsequent language information. For example, if the correction is not made, the output voice is consistent and stable in speed, without a sense of frustration, making it difficult for users to obtain a real interactive experience.

[0136] Furthermore, the distribution parameters to be corrected are corrected using a second loss function to obtain the prior phoneme feature distribution parameters carrying the predicted duration. Specifically, the distribution parameters to be corrected are processed to obtain an alignment matrix; the alignment matrix is ​​converted into an unsupervised duration; and the predicted duration is obtained based on the encoded phoneme vector. The unsupervised duration and the predicted duration are input into the second loss function to calculate the corresponding second loss value; when the second loss value does not meet the second loss threshold, the distribution parameters to be corrected are corrected based on the alignment matrix; when the second loss value meets the second loss threshold, the correction is terminated to obtain the prior phoneme feature distribution parameters.

[0137] Specifically, first of all, we need to clarify the role of the parameter μ of the prior Mel spectrum distribution in speech synthesis. The parameter μ usually represents the distribution characteristics of phonemes on the Mel spectrum, which is crucial for generating natural and fluent speech. In order to obtain the precise correspondence between phonemes and Mel spectrum, the Monotonic_Align_Search algorithm (MAS) is used to further process the parameter μ.

[0138] Next, using the MAS algorithm, the alignment information between the phoneme and the Mel spectrum can be extracted from the parameter μ and an alignment matrix (Aligner_Matrix) can be generated. The elements in this alignment matrix represent the correspondence between the phoneme and the Mel spectrum frame. Through the alignment matrix, the starting and ending positions of each phoneme on the Mel spectrum can be clearly known, thereby obtaining the unsupervised phoneme duration (Duration_).

[0139] Then, the encoded phoneme vector is input into the duration prediction component, which predicts the duration of the phoneme based on a neural network model (such as a recurrent neural network, a convolutional neural network, etc.). There may be a difference between the predicted duration Duration and the unsupervised duration (Duration_), so the loss value between them needs to be calculated.

[0140] In order to evaluate the difference between the predicted duration and the unsupervised duration, the second loss function is used to calculate the corresponding second loss value. The second loss function can be designed according to the specific task requirements. Common loss functions include mean square error loss, cross entropy loss, etc. By calculating the second loss value, the degree of inconsistency between the predicted duration and the unsupervised duration can be quantified.

[0141] Next, check whether the second loss value meets the preset second loss threshold. If the second loss value does not meet the threshold requirement, it means that the difference between the predicted duration and the unsupervised duration is large and needs to be corrected. At this time, the distribution parameters to be corrected are corrected based on the alignment matrix. The correction method can be designed according to the specific task requirements. For example, the difference between the predicted duration and the unsupervised duration can be reduced by adjusting the weight of the duration prediction component, optimizing the network structure, etc.

[0142] Finally, when the second loss value meets the second loss threshold, it means that the difference between the predicted duration and the unsupervised duration is small enough, and the correction process can be terminated. At this point, the obtained prior phoneme feature distribution parameters are relatively accurate and can be used in subsequent speech synthesis tasks.

[0143] In summary, by using the MAS algorithm to obtain the alignment matrix, we can then obtain the unsupervised supervision signal of the duration of the phoneme, which can be compared with the predicted duration and the loss value can be calculated. According to the size of the loss value, necessary corrections can be made to the distribution parameters to be corrected to obtain more accurate prior phoneme feature distribution parameters. This process helps to improve the quality and naturalness of speech synthesis.

[0144] Based on the same idea, the present application embodiment also provides a speech generation method. Figure 5 A flow chart of a speech generation method provided in an embodiment of the present application. Figure 5 As can be seen in , the method includes the following steps:

[0145] Step 501: Obtain the embedded representation of the text information to be processed and the target object sound.

[0146] Step 502: Convert the text information into a target phoneme feature representation.

[0147] Step 503: Generate prior phoneme feature distribution parameters that carry target object sound characteristics based on the target phoneme feature representation and the embedded representation of the target object sound.

[0148] Step 504: Input the target object index corresponding to the prior phoneme feature distribution parameter and the embedded representation of the target object sound into the trained second model, and output speech information integrated with the target object sound characteristics.

[0149] For ease of understanding, the following will be described by way of specific embodiments. Figure 6 The following is a flow chart of speech generation for illustrating an embodiment of the present application. Figure 6 As can be seen in the figure, when using the model obtained by the above training, only the text information to be processed (the text information needs to be converted into the target phoneme feature representation before being input into the model) and the embedded representation of the target object sound (each target object sound can be uniquely represented by the corresponding target object ID, which can be searched by ID when used) are needed, so that the target phoneme feature representation and the embedded representation of the target object sound can be spliced ​​later. In practical applications, the embedded representation of the target object sound can be stored in a database, selected and used from the database as needed. When storing, it can be stored in a key-value pair, and each embedded representation of the target object sound has its unique corresponding spk_idx. The user only needs to specify or input spk_idx into the model, and the corresponding embedded representation of the target object sound will be found from the database according to spk_idx when the model is executed later. A large number of embedded representations of the target object sound are stored in the database, so as to meet the speech synthesis requirements of different virtual characters and realize that different virtual characters have different natural speech synthesis and expression capabilities.

[0150] Next, the step 503 is performed to generate target priori phoneme feature distribution parameters carrying the target object sound characteristics according to the target phoneme feature representation and the embedded representation of the target object sound, including:

[0151] Step 5031: The target phoneme feature representation is fused with the embedded representation of the target object sound and input into the encoder to generate a third phoneme vector.

[0152] Step 5032: Determine the predicted phoneme duration based on the third phoneme vector.

[0153] Step 5033: Process the third phoneme vector to obtain the prior phoneme feature distribution parameters to be corrected.

[0154] Step 5034: Use the predicted phoneme duration to correct the prior phoneme feature distribution parameters to be corrected, and generate prior phoneme feature distribution parameters that carry the sound characteristics of the target object.

[0155] For example, if Figure 6 As shown, after the target phoneme feature representation is fused with the embedded representation of the target object sound and input into the encoder, a third phoneme vector is output after encoding processing.

[0156] Further, the third phoneme vector is input into the convolution layer and the duration prediction component respectively. After being input into the convolution layer, the prior phoneme feature distribution parameter to be corrected is output. After being input into the duration prediction component, the predicted phoneme duration is output, and the predicted phoneme duration is used to correct the prior phoneme feature distribution parameter to be corrected, so that the target prior phoneme feature distribution parameter (Prior_melμ) obtained after correction can better reflect the sound characteristics of the target object, such as the speech speed, intonation, timbre, etc. that are unique to the target object.

[0157] Further, as described in step 504, the target priori phoneme feature distribution parameter and the target object index corresponding to the embedded representation of the target object sound are input into the trained diffusion model, and the speech information fused with the target object sound characteristics is output, including:

[0158] Step 5041: Input the prior phoneme feature distribution parameters into the second model for forward diffusion processing to obtain target noise distribution parameters containing noise information.

[0159] Step 5042: Perform reverse denoising on the target noise distribution parameters using the target object index corresponding to the embedded representation of the target object sound, to obtain speech information integrated with the target object sound characteristics.

[0160] The second model mentioned here is the ability to learn to generate speech that incorporates the sound characteristics of the target object.

[0161] The spk_emb mentioned here usually refers to the embedded representation of the target object's voice (Speaker Embedding). In tasks such as speech processing, speech recognition, or speech synthesis, in order to distinguish different target objects, we usually use the embedding of the target object's voice to represent the characteristics of the target object. This embedding is usually a fixed-dimensional vector extracted from the target object's voice sample by training the model. This vector can capture the individual characteristics of the target object, allowing the model to take the target object's information into account when processing speech.

[0162] Specifically, in speech synthesis, spk_emb can be used to control the identity of the target object of the generated speech, so that the synthesized speech has specific target object characteristics. In speech recognition, it can help the model better recognize the speech of different target objects and improve the recognition accuracy.

[0163] In short, spk_emb is an embedding vector used to represent the characteristics of the target object, which plays an important role in speech processing related tasks.

[0164] In the above manner, when synthesizing speech, an embedded representation of the target object's sound is provided while inputting text information. The embedded representation of the target object's sound is involved in multiple calculation links, so that the speech finally synthesized by the model incorporates the target object's sound characteristics specified by the user. At the same time, the second model is used as the student model. While learning from the first model the ability to generate speech information containing the target object's sound characteristics, the second model is smaller than the first model. The second model can retain the performance of the first model while reducing the consumption of computing resources.

[0165] Based on the same idea, the present application embodiment also provides a model training device. Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present application. Figure 7 As can be seen, the device includes:

[0166] The generating module 71 is used to generate a priori phoneme feature distribution parameters carrying the target object sound characteristics according to the phoneme feature representation corresponding to the target text and the embedded representation of the target object sound.

[0167] The generation module 71 mentioned here includes PhonemeEmbedding (phoneme steering) for converting a phoneme sequence into a vector, and Prenet (neural network) for fusing the vector with the embedded representation of the target object sound, and the phoneme feature representation is obtained after fusion. It also includes an encoder for encoding the phoneme feature representation, and a convolution layer for calculating the prior phoneme feature distribution parameters (that is, the prior Mel spectrum distribution parameters μ). In addition, it also includes a duration prediction component for correcting the prior phoneme feature distribution parameters.

[0168] The training module 72 is used to input the embedded representation of the target object sound and the prior phoneme feature distribution parameters into the first model for knowledge distillation training to obtain a second model; wherein the second model learns from the first model the ability to generate speech prediction by integrating the target object sound characteristics.

[0169] The training module 72 mentioned here includes a first model (i.e., teacher model) and a second model (i.e., student model). In the training phase, the teacher model is trained using the prepared training samples, and then the student model is made to learn the ability to generate the predicted speech by integrating the sound characteristics of the target object using the consistency distillation method.

[0170] Optionally, the training module 72 is used to input the priori phoneme feature distribution parameters into the first model for forward diffusion processing to obtain noise distribution parameters including noise information;

[0171] Sampling the noise distribution parameters to obtain a sampling trajectory;

[0172] Performing reverse denoising processing on the sampling trajectory using the first solver and the embedded representation of the target object sound to obtain a predicted sample incorporating the target object sound characteristics;

[0173] When it is determined that the predicted sample meets the requirements, consistent knowledge distillation training is performed to obtain the second model.

[0174] Optionally, a training module 73 is used to obtain a trained first model when determining that the predicted sample meets the requirement using the first loss function;

[0175] The first model is used to perform consistency knowledge distillation training on the model to be trained, so as to obtain the second model that learns from the first model the ability to generate speech prediction by integrating the sound characteristics of the target object starting from any point in the sampling trajectory.

[0176] Optionally, a generating module 72 is used to convert the phoneme feature representation into a first phoneme vector;

[0177] Concatenating the first phoneme vector with the embedded representation of the target object sound to obtain a second phoneme vector;

[0178] Inputting the second phoneme vector into an encoder to generate an encoded phoneme vector;

[0179] The encoded phoneme vector is processed to obtain the prior phoneme feature distribution parameter carrying the predicted duration.

[0180] Optionally, a generating module 72 is used to process the encoded phoneme vector to obtain a distribution parameter to be corrected;

[0181] The distribution parameters to be corrected are corrected using a second loss function to obtain the corrected prior phoneme feature distribution parameters carrying the predicted duration.

[0182] Optionally, a generating module 72 is used to process the distribution parameters to be corrected to obtain an alignment matrix;

[0183] Converting the alignment matrix to unsupervised duration;

[0184] Determining a predicted duration according to the encoded phoneme vector;

[0185] Input the unsupervised duration and the predicted duration into the second loss function to calculate the corresponding second loss value;

[0186] When the second loss value does not meet the second loss threshold, correcting the to-be-corrected distribution parameter based on the alignment matrix;

[0187] When the second loss value meets the second loss threshold, the correction is terminated to obtain the prior phoneme feature distribution parameter carrying the predicted duration.

[0188] Based on the same idea, the present application also provides a speech generating device. Figure 8 A schematic diagram of the structure of a speech generating device provided in an embodiment of the present application. Figure 8 As can be seen, the device includes:

[0189] The acquisition module 81 is used to acquire the embedded representation of the text information to be processed and the target object sound.

[0190] The conversion module 82 is used to convert the text information into a target phoneme feature representation.

[0191] The conversion module 82 mentioned here includes Phoneme Embedding (phoneme transformation quantity) for converting a phoneme sequence into a vector, and Prenet (neural network) for fusing the vector with the embedded representation of the target object sound, and the phoneme feature representation is obtained after the fusion.

[0192] The generating module 83 is used to generate a priori phoneme feature distribution parameters carrying the target object sound characteristics according to the target phoneme feature representation and the embedded representation of the target object sound.

[0193] The generation module 83 mentioned here includes an encoder for encoding the phoneme feature representation, and a convolution layer for calculating the prior phoneme feature distribution parameter (that is, the prior Mel spectrum distribution parameter μ). In addition, it also includes a duration prediction component for correcting the prior phoneme feature distribution parameter.

[0194] The output module 84 is used to input the target object index corresponding to the prior phoneme feature distribution parameter and the embedded representation of the target object sound into the trained second model, and output the speech information integrated with the target object sound characteristics.

[0195] A generating module 83, configured to input the target phoneme feature representation fused with the embedded representation of the target object sound into the encoder to generate a third phoneme vector;

[0196] Inputting the third phoneme vector into a duration prediction component to obtain a predicted phoneme duration;

[0197] Inputting the third phoneme vector into the convolution layer to obtain a priori phoneme feature distribution parameter to be corrected;

[0198] The predicted phoneme duration is used to correct the prior phoneme feature distribution parameters to be corrected, so as to generate prior phoneme feature distribution parameters carrying the sound characteristics of the target object.

[0199] An output module 84 is used to input the target priori phoneme feature distribution parameter into the second model for forward diffusion processing to obtain a target noise distribution parameter containing noise information;

[0200] The target noise distribution parameters are subjected to reverse denoising processing using the target object index corresponding to the embedded representation of the target object sound, so as to obtain speech information integrated with the target object sound characteristics.

[0201] In one possible design, the above Figure 7 The model training device shown, Figure 8 The structure of the speech generating device shown can be implemented as an electronic device. Fig. 9 As shown, the electronic device may include: a processor 91 and a memory 92. The memory 92 stores executable code, and when the executable code is executed by the processor 91, the processor 91 can at least implement the lip animation generation method provided in the above embodiment. The structure of the electronic device may also include a communication interface 93 for communicating with other devices or communication networks.

[0202] In addition, an embodiment of the present invention provides a non-transitory machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor in a server or a computer, the processor executes the operations provided in the above embodiments. Figure 1 The corresponding model training method and Figure 5 The corresponding speech generation method.

[0203] The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0204] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution can essentially or contribute to the part in the form of a computer product, and the present invention can be implemented in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0205] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model training method, It is characterized in that The method comprises: Generate a priori phoneme feature distribution parameters carrying the sound characteristics of the target object according to the phoneme feature representation corresponding to the target text and the embedded representation of the target object sound; The embedded representation of the target object sound and the prior phoneme feature distribution parameters are input into the first model for consistency knowledge distillation training to obtain a second model; wherein the second model learns from the first model the ability to generate speech predicted by integrating the target object sound characteristics.

2. The method according to claim 1, It is characterized in that The step of inputting the embedded representation of the target object sound and the prior phoneme feature distribution parameters into the first model for consistent knowledge distillation training to obtain the second model comprises: Inputting the priori phoneme feature distribution parameters into the first model for forward diffusion processing to obtain noise distribution parameters including noise information; Sampling the noise distribution parameters to obtain a sampling trajectory; Performing reverse denoising processing on the sampling trajectory using the first solver and the embedded representation of the target object sound to obtain a predicted sample incorporating the target object sound characteristics; When it is determined that the predicted sample meets the requirements, consistent knowledge distillation training is performed to obtain the second model.

3. The method according to claim 2, It is characterized in that When it is determined that the predicted sample meets the requirement, performing consistency knowledge distillation training to obtain the second model includes: When the first loss function is used to determine that the predicted sample meets the requirement, a first model that has been trained is obtained; The first model is used to perform consistency knowledge distillation training on the model to be trained, so as to obtain the second model that learns from the first model the ability to generate speech prediction by integrating the sound characteristics of the target object starting from any point in the sampling trajectory.

4. The method according to claim 1, It is characterized in that The generating of a priori phoneme feature distribution parameters carrying the sound characteristics of the target object according to the phoneme feature representation corresponding to the target text and the embedded representation of the target object sound comprises: Converting the phoneme feature representation into a first phoneme vector; Concatenating the first phoneme vector with the embedded representation of the target object sound to obtain a second phoneme vector; Inputting the second phoneme vector into an encoder to generate an encoded phoneme vector; The encoded phoneme vector is processed to obtain the prior phoneme feature distribution parameter carrying the predicted duration.

5. The method according to claim 4, It is characterized in that The processing of the encoded phoneme vector to obtain the prior phoneme feature distribution parameter carrying the predicted duration includes: Processing the encoded phoneme vector to obtain a distribution parameter to be corrected; The distribution parameters to be corrected are corrected using a second loss function to obtain the prior phoneme feature distribution parameters carrying the predicted duration.

6. The method according to claim 5, It is characterized in that The method of correcting the to-be-corrected distribution parameter by using the second loss function to obtain the priori phoneme feature distribution parameter carrying the predicted duration includes: Processing the distribution parameters to be corrected to obtain an alignment matrix; Converting the alignment matrix to unsupervised duration; Determining a predicted duration according to the encoded phoneme vector; Input the unsupervised duration and the predicted duration into the second loss function to calculate the corresponding second loss value; When the second loss value does not meet the second loss threshold, correcting the to-be-corrected distribution parameter based on the alignment matrix; When the second loss value meets the second loss threshold, the correction is terminated to obtain the prior phoneme feature distribution parameter carrying the predicted duration.

7. A method for generating speech, It is characterized in that The method comprises: Obtaining embedded representations of the text information to be processed and the target object's sound; Converting the text information into a target phoneme feature representation; Generate a priori phoneme feature distribution parameters carrying the target object sound characteristics according to the target phoneme feature representation and the embedded representation of the target object sound; The target object index corresponding to the prior phoneme feature distribution parameter and the embedded representation of the target object sound is input into a trained second model, and speech information fused with the target object sound characteristics is output.

8. The method according to claim 7, It is characterized in that The step of generating a priori phoneme feature distribution parameters carrying the target object sound characteristics according to the target phoneme feature representation and the embedded representation of the target object sound comprises: Inputting the target phoneme feature representation fused with the embedded representation of the target object sound into an encoder to generate a third phoneme vector; Determine a predicted phoneme duration according to the third phoneme vector; Processing the third phoneme vector to obtain a priori phoneme feature distribution parameter to be corrected; The predicted phoneme duration is used to correct the prior phoneme feature distribution parameters to be corrected, so as to generate prior phoneme feature distribution parameters carrying the sound characteristics of the target object.

9. The method according to claim 7, It is characterized in that The step of inputting the target object index corresponding to the prior phoneme feature distribution parameter and the embedded representation of the target object sound into a trained second model, and outputting speech information fused with the target object sound characteristics, comprises: Inputting the prior phoneme feature distribution parameter into the second model for forward diffusion processing to obtain a target noise distribution parameter containing noise information; The target noise distribution parameters are subjected to reverse denoising processing using the target object index corresponding to the embedded representation of the target object sound, so as to obtain speech information integrated with the target object sound characteristics.

10. A model training device, It is characterized in that The device comprises: A generation module, used to generate a priori phoneme feature distribution parameters carrying the sound characteristics of the target object according to the phoneme feature representation corresponding to the target text and the embedded representation of the target object sound; A training module is used to input the embedded representation of the target object sound and the prior phoneme feature distribution parameters into the first model for consistency knowledge distillation training to obtain a second model; wherein the second model learns from the first model the ability to generate speech prediction that integrates the target object sound characteristics.

11. A speech generating device, It is characterized in that The device comprises: An acquisition module, used to acquire the text information to be processed and the embedded representation of the target object sound; A conversion module, used for converting the text information into a target phoneme feature representation; A generating module, configured to generate a priori phoneme feature distribution parameters carrying the target object sound characteristics according to the target phoneme feature representation and the embedded representation of the target object sound; The output module is used to input the target object index corresponding to the prior phoneme feature distribution parameter and the embedded representation of the target object sound into the trained second model, and output the speech information integrated with the target object sound characteristics.

12. An electronic device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 9.

13. A computer-readable medium having stored thereon at least one instruction, at least one program, code set or instruction set, wherein the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the method according to any one of claims 1 to 9.

Citation Information

Cited By

  • Multi-dialect speech recognition method and device based on synthetic speech data

    CN119380701A

  • Fast text guidance sound effect generation method and system based on segmented rectification

    CN119811365A

  • Phoneme alignment method and device, equipment and medium

    CN119832928A

  • Phoneme alignment method, device, equipment and medium

    CN119832928B

  • Take-out rider intelligent helmet voice recognition method and system

    CN120319223A