Voice generation method and device fusing environment information, storage medium and equipment

Abstract

The invention discloses a voice generation method and device fusing environment information, a storage medium and equipment, relates to the technical field of sound signal processing and the technical field of finance and medical treatment, and mainly aims to solve the problem that in the prior art, voice generated based on text content of the voice lacks environment information except semantic information. Comprising the steps of obtaining a to-be-processed semantic text and a to-be-fused target environment description text, and determining multiple pieces of target voice information from a voice library based on the to-be-fused target environment description text; the target voice information carries target environment information corresponding to the to-be-fused target environment description text; performing training processing on a diffusion model based on the multiple pieces of target voice information to obtain a target diffusion model used for predicting noise included in the target environment information; and performing environment fusion processing on the to-be-processed semantic text based on the target diffusion model to obtain fused voice information fused with the target environment information.

Description

Technical Field

[0001] The present invention relates to the fields of sound signal processing technology, finance and medical technology, and in particular to a speech generation method and device, storage medium and equipment integrating environmental information. Background Art

[0002] In recent years, speech generation systems have made significant progress in the fidelity and diversity of generated speech, gradually improving the clarity and naturalness of synthesized speech. Some speech generation systems have achieved a quality level comparable to single-speaker recordings through text prompts. These speech generation systems have demonstrated their powerful ability to generate speech based on semantic information described in natural language, and are widely used in the fields of finance and medical technology.

[0003] However, the speech generation system based on text prompts still faces some problems. For example, the text content of the speech is only a part of the information contained in the speech. The speech generated only by the text content of the speech lacks information such as the environment in which the speech is made and the speaker's style. It is crucial to provide the speech generation system with information other than semantic information, which is related to whether the speech generation system can generate speech that is closer to real humans. Summary of the invention

[0004] In view of this, the present invention provides a speech generation method and apparatus, storage medium, and equipment integrating environmental information, the main purpose of which is to solve the problem in the prior art that speech generated based on text content of speech lacks environmental information other than semantic information.

[0005] According to one aspect of the present invention, a method for generating speech by integrating environmental information is provided, comprising:

[0006] Acquire the semantic text to be processed and the target environment description text to be integrated, and determine a plurality of target voice information from the voice library based on the target environment description text to be integrated; the target voice information carries the target environment information corresponding to the target environment description text to be integrated;

[0007] Training a diffusion model based on the plurality of target speech information to obtain a target diffusion model for predicting the noise mixed in the target environmental information;

[0008] Based on the target diffusion model, the semantic text to be processed is subjected to environment fusion processing to obtain fused speech information that incorporates the target environment information.

[0009] Furthermore, the step of determining a plurality of target voice information from a voice library based on the target environment description text to be integrated includes:

[0010] Performing keyword extraction processing on the description text of the target environment to be integrated into, to obtain keywords of the target environment to be integrated into;

[0011] Perform keyword matching processing on the target environment keyword to be integrated with the description text of each voice information in the voice library to obtain a keyword matching result;

[0012] The target voice information is determined based on the keyword matching result.

[0013] Furthermore, the training process of the diffusion model based on the plurality of target speech information to obtain the target diffusion model for predicting the noise mixed in the target environment information includes:

[0014] Acquire the mel spectrum of each piece of the target speech information, and perform semantic recognition processing on each piece of the target speech information to obtain semantic text corresponding to each piece of the target speech information;

[0015] The mel spectrum and the semantic text corresponding to the same target speech information form a speech-text pair to obtain multiple pairs of training samples containing the target environment information;

[0016] The diffusion model is trained based on the training samples to obtain the target diffusion model used to predict the noise included in the target environment information.

[0017] Furthermore, the training process of the diffusion model based on the training samples to obtain the target diffusion model for predicting the noise included in the target environment information includes:

[0018] Encoding the mel-spectrograms in the training samples to obtain speech potential representations corresponding to the training samples;

[0019] Obtaining a preset diffusion step, and performing a stepwise noise addition process on the speech potential representation according to the sequence of the diffusion steps to obtain a noisy speech representation;

[0020] Performing text encoding processing on the semantic text in the training sample to obtain semantic encoding features;

[0021] The speech potential representation, the noisy speech representation and the semantic coding features are input into the diffusion model for training processing to obtain the target diffusion model for predicting the noise included in the target environmental information.

[0022] Furthermore, before performing environment fusion processing on the semantic text to be processed based on the target diffusion model, the method further includes:

[0023] Determine a to-be-processed noisy speech representation from a plurality of noisy speech representations by random sampling; or,

[0024] The multiple noisy speech representations are evaluated to obtain evaluation results corresponding to the respective noisy speech representations; and a noisy speech representation to be processed is determined based on the evaluation results.

[0025] Furthermore, the performing of environment fusion processing on the semantic text to be processed based on the target diffusion model to obtain fused speech information fused with the target environment information includes:

[0026] Performing text encoding processing on the semantic text to be processed to obtain semantic encoding features to be processed;

[0027] Performing environment fusion processing on the noisy speech representation to be processed and the semantic coding feature to be processed to obtain intermediate speech information fused with the target environment information;

[0028] Based on the target diffusion model, the intermediate speech information is subjected to step-by-step noise reduction processing in the order of diffusion steps to obtain a fused speech potential representation that incorporates the target environment information;

[0029] Decoding the fused speech latent representation to obtain a fused speech mel spectrum;

[0030] The fused speech mel spectrum is subjected to speech conversion processing to obtain the fused speech information.

[0031] Furthermore, the method further comprises:

[0032] receiving user feedback information on the fused voice information;

[0033] Based on the user feedback information, the fused voice information is evaluated and processed to obtain a feedback evaluation result;

[0034] Based on the feedback evaluation result, it is determined whether to update the fused speech information into the speech database.

[0035] According to another aspect of the present invention, there is provided a speech generation device integrating environmental information, comprising:

[0036] A target determination module, used to obtain the semantic text to be processed and the target environment description text to be integrated, and determine a plurality of target voice information from the voice library based on the target environment description text to be integrated; the target voice information carries the target environment information corresponding to the target environment description text to be integrated;

[0037] A training module, used for training a diffusion model based on the plurality of target speech information to obtain a target diffusion model for predicting the noise mixed in the target environment information;

[0038] The fusion module is used to perform environment fusion processing on the semantic text to be processed based on the target diffusion model to obtain fused speech information that integrates the target environment information.

[0039] Furthermore, the target determination module is also used for:

[0040] Performing keyword extraction processing on the description text of the target environment to be integrated into, to obtain keywords of the target environment to be integrated into;

[0041] Perform keyword matching processing on the target environment keyword to be integrated with the description text of each voice information in the voice library to obtain a keyword matching result;

[0042] The target voice information is determined based on the keyword matching result.

[0043] Furthermore, the training module also includes:

[0044] An extraction unit, used to obtain the mel spectrum of each target voice information, and perform semantic recognition processing on each target voice information to obtain a semantic text corresponding to each target voice information;

[0045] A combining unit, used for combining the mel spectrum and the semantic text corresponding to the same target speech information into speech-text pairs, so as to obtain a plurality of pairs of training samples containing the target environment information;

[0046] A training unit is used to perform training processing on the diffusion model based on the training samples to obtain the target diffusion model used to predict the noise included in the target environment information.

[0047] Furthermore, the training unit is also used for:

[0048] Encoding the mel-spectrograms in the training samples to obtain speech potential representations corresponding to the training samples;

[0049] Obtaining a preset diffusion step, and performing a stepwise noise addition process on the speech potential representation according to the sequence of the diffusion steps to obtain a noisy speech representation;

[0050] Performing text encoding processing on the semantic text in the training sample to obtain semantic encoding features;

[0051] The speech potential representation, the noisy speech representation and the semantic coding features are input into the diffusion model for training processing to obtain the target diffusion model for predicting the noise included in the target environmental information.

[0052] Furthermore, the device further comprises a sampling module, and the sampling module is used for:

[0053] Determine a to-be-processed noisy speech representation from a plurality of noisy speech representations by random sampling; or,

[0054] The multiple noisy speech representations are evaluated to obtain evaluation results corresponding to the respective noisy speech representations; and a noisy speech representation to be processed is determined based on the evaluation results.

[0055] Furthermore, the fusion module is also used for:

[0056] Performing text encoding processing on the semantic text to be processed to obtain semantic encoding features to be processed;

[0057] Performing environment fusion processing on the noisy speech representation to be processed and the semantic coding feature to be processed to obtain intermediate speech information fused with the target environment information;

[0058] Based on the target diffusion model, the intermediate speech information is subjected to step-by-step noise reduction processing in the order of diffusion steps to obtain a fused speech potential representation that incorporates the target environment information;

[0059] Decoding the fused speech latent representation to obtain a fused speech mel spectrum;

[0060] The fused speech mel spectrum is subjected to speech conversion processing to obtain the fused speech information.

[0061] Furthermore, the device also includes a voice library updating module, which is used to:

[0062] receiving user feedback information on the fused voice information;

[0063] Based on the user feedback information, the fused voice information is evaluated and processed to obtain a feedback evaluation result;

[0064] Based on the feedback evaluation result, it is determined whether to update the fused speech information into the speech database.

[0065] According to another aspect of the present invention, a storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to execute operations corresponding to the above-mentioned method for generating speech by integrating environmental information.

[0066] According to another aspect of the present invention, there is provided a device, comprising a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0067] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned speech generation method integrating environmental information.

[0068] By means of the above technical solution, the technical solution provided by the embodiment of the present invention has at least the following advantages:

[0069] The present invention provides a speech generation method and device, storage medium, and equipment that integrate environmental information. Compared with the prior art, the present invention uses target speech information that carries target environmental information to train a diffusion model, and obtains a target diffusion model for predicting noise mixed in the target environmental information, thereby achieving training on a large amount of real audio that carries environmental information, and improving the naturalness and fluency of the integrated speech information. The generation of speech is controlled by two natural language prompts, namely, a description text of the target environment to be integrated and a semantic text to be processed, without the need for other prompt information. With the control of the description text of the target environment to be integrated, compared with other speech generation systems in the current financial and medical fields that use text to control the emotion, volume, gender, and other factors of the generated speech, the method proposed by the present invention can generate a variety of sounds in different environments, such as speech with sound effects, singing, whispering, etc., further improving the controllability and diversity of the speech generation system.

[0070] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0072] Figure 1 A schematic flow chart of a method for generating speech by integrating environmental information provided by an embodiment of the present invention is shown;

[0073] Figure 2 A schematic flow chart of another method for generating speech by integrating environmental information provided by an embodiment of the present invention is shown;

[0074] Figure 3 A schematic flow chart of another method for generating speech by integrating environmental information provided by an embodiment of the present invention is shown;

[0075] Figure 4 A schematic flow chart of another method for generating speech by integrating environmental information provided by an embodiment of the present invention is shown;

[0076] Figure 5 A schematic diagram showing the structure of a speech generation device integrating environmental information provided by an embodiment of the present invention;

[0077] Figure 6 A schematic structural diagram of a device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0078] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0079] The embodiment of the present invention provides a method for generating speech by integrating environmental information. Figure 1 As shown, the method includes:

[0080] 101. Obtain a semantic text to be processed and a target environment description text to be integrated, and determine a plurality of target voice information from a voice library based on the target environment description text to be integrated; the target voice information carries target environment information corresponding to the target environment description text to be integrated;

[0081] In an embodiment of the present invention, the current execution end obtains a semantic text to be processed and a description text of the target environment to be integrated. Among them, the semantic text to be processed contains semantic information for generating fused speech, and the description text of the target environment to be integrated represents the description information of the integrated environment, such as "low noise on the street", "flowing water and bird singing in the mountains", etc., which is not specifically limited in the embodiment of the present invention. The current execution end determines multiple target voice information from the voice library based on the description text of the target environment to be integrated. Among them, the voice library stores voice information carrying various environmental information, such as voice information of conversations on the street, voice information of conversations in the valley, voice information of whispering, etc., which is not specifically limited in the embodiment of the present invention. The target voice information determined in the embodiment of the present invention carries the target environment information corresponding to the description text of the target environment to be integrated. For example, when the description text of the target environment to be integrated is "low noise on the street", the voice information of conversations on the street is determined as the target voice information, etc., which is not specifically limited in the embodiment of the present invention.

[0082] 102. Training a diffusion model based on the plurality of target speech information to obtain a target diffusion model for predicting noise mixed in the target environmental information;

[0083] In an embodiment of the present invention, the current execution end trains the diffusion model based on multiple target voice information to obtain a target diffusion model for predicting the mixed noise in the target environment information. Among them, the diffusion model (DenoisingDiffusionProbabilistic Model) is referred to as DDPM, which includes a forward diffusion process and a reverse diffusion process. In an embodiment of the present invention, the forward diffusion process of the diffusion model is to perform layer-by-layer noise processing on the target voice information, and the reverse diffusion process is to perform layer-by-layer denoising processing on the voice information after the noise is added, so as to restore the target voice information. Through the forward diffusion process and the reverse diffusion process, the diffusion model can be made to have the ability to recognize the mixed noise in the target environment information, thereby obtaining a target diffusion model for predicting the mixed noise in the target environment.

[0084] 103. Performing environment fusion processing on the semantic text to be processed based on the target diffusion model to obtain fused speech information that incorporates the target environment information.

[0085] In the embodiment of the present invention, since the target diffusion model trained in step 102 has the ability to identify the noise mixed in the target environmental information, the current execution end can remove the noise in the voice information after environmental fusion based on the target diffusion model trained in step 102. Specifically, during environmental fusion processing, it is necessary to first obtain the intermediate voice information mixed with noise and fused with the target environmental information based on the semantic text to be processed, and then process the noise in the intermediate voice information fused with the target environmental information based on the target diffusion model to obtain the fused voice information fused with the target environmental information.

[0086] It should be noted that the method of the present invention can be applied to human-computer interaction scenarios, product promotion scenarios, etc. in the financial and medical fields, and the embodiments of the present invention are not specifically limited.

[0087] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to improve the noise recognition ability of the diffusion model, another speech generation method integrating environmental information is provided, such as Figure 2 As shown, the step of determining multiple target voice information from the voice library based on the target environment description text to be integrated includes:

[0088] 201. Perform keyword extraction processing on the description text of the target environment to be integrated into, to obtain keywords of the target environment to be integrated into;

[0089] In an embodiment of the present invention, the current execution end performs keyword extraction on the description text of the target environment to be integrated to obtain keywords of the target environment to be integrated. For example, when the description text of the target environment to be integrated is "low noise on the street", the keywords of the target environment to be integrated are "street" and "low noise"; when the description text of the target environment to be integrated is "flowing water and birds singing in the mountains", the keywords of the target environment to be integrated are "mountains", "flowing water" and "birds singing", etc., which are not specifically limited in the embodiment of the present invention.

[0090] 202, performing keyword matching processing on the keyword to be integrated into the target environment and the description text of each voice information in the voice database to obtain a keyword matching result;

[0091] In an embodiment of the present invention, the current execution end performs keyword matching processing on the target environment keywords to be integrated obtained in step 201 and the description texts of each voice information in the voice library to obtain a keyword matching result of successful matching or failed matching. For example, the voice library contains voice information of conversations on the street, voice information of conversations in the valley, voice information of whispering, etc., and the above voice information all have corresponding description texts. When keyword matching processing is performed on the target environment keywords "street" and "low noise" to be integrated in step 201 and the description texts of each voice information in the voice library, the description text corresponding to the voice information of conversations on the street in the voice library can be successfully matched; the description text corresponding to the voice information of conversations in the valley in the voice library cannot be matched; the description text corresponding to the voice information of whispering in the voice library cannot be matched, etc., and the embodiment of the present invention does not make specific limitations.

[0092] 203. Determine the target voice information based on the keyword matching result.

[0093] In the embodiment of the present invention, the current execution end determines the successfully matched voice information as the target voice information based on the keyword matching result obtained in step 202, which is used for the subsequent training process of the diffusion model.

[0094] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to facilitate the training of the diffusion model and improve the efficiency of model training, another speech generation method integrating environmental information is provided, such as Figure 3 As shown, the step of training the diffusion model based on the plurality of target speech information to obtain a target diffusion model for predicting the mixed noise in the target environment information includes:

[0095] 301. Acquire the mel spectrum of each piece of target voice information, and perform semantic recognition processing on each piece of target voice information to obtain a semantic text corresponding to each piece of target voice information;

[0096] In an embodiment of the present invention, the current execution end obtains the mel-spectrogram of each target voice information. The mel-spectrogram combines the time domain characteristics and frequency domain characteristics of the voice information, and shows the change of the voice frequency over time. In addition, the current execution end also performs semantic recognition processing on each target voice information respectively to obtain the semantic text corresponding to each target voice information. The semantic recognition processing can be completed by using the automatic speech recognition (Automatic Speech Recognition, ASR) technology, which is not specifically limited in the embodiment of the present invention.

[0097] 302. Combining the mel-spectrogram and the semantic text corresponding to the same target speech information into speech-text pairs, and obtaining a plurality of pairs of training samples containing the target environment information;

[0098] In the embodiment of the present invention, the current execution end projects the information of the two modalities of mel spectrum and semantic text corresponding to the same target speech information into the same latent space to form speech-text pairs, denoted as <speech, text>, thereby obtaining multiple pairs of training samples containing target environment information. Among them, the CLAP (Contrastive Language-Audio Pretraining) model can be used to process the projection of the two modalities of mel spectrum and semantic text. The core of the CLAP model lies in its unique pre-training model, which can extract potential representations from any given audio and text, denoted as c e , suitable for various downstream tasks.

[0099] 303. Perform training processing on a diffusion model based on the training samples to obtain the target diffusion model for predicting noise included in the target environment information.

[0100] In an embodiment of the present invention, the current execution end performs training processing on the diffusion model based on the training samples to obtain a target diffusion model for predicting the noise included in the target environment information. Specifically, the following steps are included:

[0101] (1) Encoding the mel-spectrograms in the training samples to obtain a speech potential representation corresponding to each training sample, denoted as z0; wherein the encoding process can be implemented by a variational autoencoder (VAE), which is not specifically limited in the embodiment of the present invention.

[0102] (2) Obtaining a preset diffusion step, and performing a stepwise noise addition process on the speech latent representation according to the sequence of the diffusion steps to obtain a noisy speech representation; for example, performing a first round of noise addition process on the speech latent representation z0 to obtain a first noisy speech representation z1; then performing a second round of noise addition process on the first noisy speech representation to obtain a second noisy speech representation z2; then performing a third round of noise addition process on the second noisy speech representation to obtain a third noisy speech representation z3, ...; then performing a t-th round of noise addition process on the t-1th noisy speech representation to obtain a t-th noisy speech representation z t ...until all the diffusion steps are completed, and the noisy speech representation z after the Tth round of noisy processing is obtained T It should be noted that the noisy speech after the Tth round of noise processing is represented by z T It becomes pure Gaussian noise that carries the target environment information.

[0103] (3) Performing text encoding processing on the semantic text in the training sample to obtain a semantic encoding feature, denoted as c; wherein the text encoding processing is used to convert the semantic text into a high-dimensional vector, and an existing text encoder can be selected to achieve this, and the embodiment of the present invention does not make any specific limitation.

[0104] (4) The above-mentioned speech potential representation z0 and the noisy speech representation z t The target diffusion model is trained by inputting the semantic coding feature c into the diffusion model to obtain the target diffusion model used to predict the noise included in the target environment information. The target diffusion model predicts z t The noise added in is used to obtain z0, and the training objective is:

[0105]

[0106] Where ε is z t The noise added in θ is the predicted noise, c e Latent representations extracted from audio and text.

[0107] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to improve the fusion efficiency and fusion effect, another speech generation method integrating environmental information is provided, wherein before performing environmental fusion processing on the semantic text to be processed based on the target diffusion model, the method further includes:

[0108] Determine a to-be-processed noisy speech representation from a plurality of noisy speech representations by random sampling; or,

[0109] The multiple noisy speech representations are evaluated to obtain evaluation results corresponding to the respective noisy speech representations; and a noisy speech representation to be processed is determined based on the evaluation results.

[0110] In the embodiment of the present invention, the current execution end can select from multiple noisy speech representations z by random sampling. T Determine a noisy speech representation z to be processed T '; Alternatively, the current execution end may also evaluate multiple noisy speech representations to obtain the corresponding noisy speech representations z T Corresponding evaluation results. For example, based on the ability of the diffusion model to restore the speech potential representation z0 during training, the restoration ability is sorted from high to low, and the best restored noisy speech representation is determined as the noisy speech representation to be processed z T ', etc., which are not specifically limited in the embodiments of the present invention.

[0111] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to more accurately obtain fused speech information fused with target environment information, another speech generation method fused with environment information is provided, such as Figure 4 As shown, the step performs environment fusion processing on the semantic text to be processed based on the target diffusion model to obtain fused speech information that integrates the target environment information, including:

[0112] 401. Perform text encoding processing on the semantic text to be processed to obtain semantic encoding features to be processed;

[0113] In the embodiment of the present invention, the current execution end performs text encoding processing on the semantic text to be processed to obtain the semantic encoding feature to be processed, which is recorded as c'. The text encoding processing is used to convert the semantic text to be processed into a high-dimensional vector, which can be implemented by selecting an existing text encoder, and the embodiment of the present invention does not make any specific limitation.

[0114] 402, performing environment fusion processing on the noisy speech representation to be processed and the semantic coding feature to be processed to obtain intermediate speech information fused with the target environment information;

[0115] In the embodiment of the present invention, the current execution end represents the noisy speech to be processed as z T ' and the semantic coding feature c' to be processed obtained in step 401 are subjected to environment fusion processing to obtain the intermediate speech information fused with the target environment information; it should be noted that since the noisy speech representation z T ' represents pure Gaussian noise carrying target environment information, so the process of environment fusion processing in the embodiment of the present invention is the process of adding noise.

[0116] 403. Based on the target diffusion model, the intermediate speech information is subjected to step-by-step noise reduction processing in the order of diffusion steps to obtain a fused speech potential representation that incorporates the target environment information;

[0117] In the embodiment of the present invention, the current execution end performs stepwise noise reduction processing on the intermediate speech information obtained in step 402 according to the order of diffusion steps based on the target diffusion model obtained by training from steps 301 to 303, that is, executes the inverse diffusion process of the diffusion model, thereby obtaining the fused speech potential representation z0' that integrates the target environment information.

[0118] 404. Decode the fused speech potential representation to obtain a fused speech mel spectrum.

[0119] In the embodiment of the present invention, the current execution end decodes the fused speech potential representation z0' obtained in step 403 to obtain the fused speech mel spectrum. The decoding process is the reverse operation process of the encoding process recorded in step 303. In the embodiment of the present invention, a variational autoencoder (VAE) is preferably used to implement it, and the embodiment of the present invention does not make any specific limitation.

[0120] 405. Perform speech conversion processing on the fused speech mel-spectrogram to obtain the fused speech information.

[0121] In the embodiment of the present invention, the current execution end performs speech conversion processing on the fused speech mel spectrum obtained in step 404, thereby obtaining fused speech information that incorporates target environment information. The speech conversion processing can be implemented by selecting a vocoder, which is not specifically limited in the embodiment of the present invention.

[0122] Furthermore, as a refinement and extension of the specific implementation of the above embodiment, in order to further improve the effect of environment fusion, another method for generating speech by integrating environment information is provided, and the method further includes:

[0123] receiving user feedback information on the fused voice information;

[0124] In the embodiment of the present invention, the current execution end receives user feedback information on the fused voice information obtained from step 401 to step 405; for example, the user's evaluation of the fused voice information or the user's evaluation score of the fused voice information, etc., which is not specifically limited in the embodiment of the present invention.

[0125] Based on the user feedback information, the fused voice information is evaluated and processed to obtain a feedback evaluation result;

[0126] Based on the feedback evaluation result, it is determined whether to update the fused speech information into the speech database.

[0127] In an embodiment of the present invention, the current execution end evaluates and processes the obtained fused voice information based on the user feedback information to obtain a feedback evaluation result. For example, the user satisfaction is evaluated through the user's evaluation to obtain a feedback evaluation result of whether the user is satisfied; or, the user's score is compared with a preset score threshold to obtain a feedback evaluation result of whether the score threshold is exceeded, etc., which is not specifically limited in the embodiment of the present invention. Finally, based on the feedback evaluation result, determine whether to update the fused voice information to the voice library. For example, the fused voice information corresponding to the above-mentioned user satisfied feedback evaluation result is updated as a new sample to the voice library; or, the fused voice information corresponding to the above-mentioned feedback evaluation result that exceeds the score threshold is updated as a new sample to the voice library, etc., which is not specifically limited in the embodiment of the present invention.

[0128] The embodiment of the present invention provides a method for generating speech that integrates environmental information. Compared with the prior art, the present invention uses target speech information that carries target environmental information to train a diffusion model, and obtains a target diffusion model for predicting noise mixed in the target environmental information, thereby achieving training on a large amount of real audio that carries environmental information, and improving the naturalness and fluency of the integrated speech information. The generation of speech is controlled by two natural language prompts, the target environment description text to be integrated and the semantic text to be processed, without the need for other prompt information. With the control of the target environment description text to be integrated, compared with other speech generation systems in the current financial and medical fields that use text to control the emotion, volume, gender and other factors of the generated speech, the method proposed by the present invention can generate a variety of sounds in different environments, such as speech with sound effects, singing, whispering, etc., further improving the controllability and diversity of the speech generation system.

[0129] As the above Figure 1 The implementation of the method shown in the figure, the embodiment of the present invention provides a speech generation device integrating environmental information, such as Figure 5 As shown, the device comprises:

[0130] The target determination module 51 is used to obtain the semantic text to be processed and the target environment description text to be integrated, and determine a plurality of target voice information from the voice library based on the target environment description text to be integrated; the target voice information carries the target environment information corresponding to the target environment description text to be integrated;

[0131] A training module 52 is used to train a diffusion model based on the plurality of target speech information to obtain a target diffusion model for predicting the noise mixed in the target environment information;

[0132] The fusion module 53 is used to perform environment fusion processing on the semantic text to be processed based on the target diffusion model to obtain fused speech information that integrates the target environment information.

[0133] Furthermore, the target determination module 51 is also used for:

[0134] Performing keyword extraction processing on the description text of the target environment to be integrated into, to obtain keywords of the target environment to be integrated into;

[0135] Perform keyword matching processing on the target environment keyword to be integrated with the description text of each voice information in the voice library to obtain a keyword matching result;

[0136] The target voice information is determined based on the keyword matching result.

[0137] Furthermore, the training module 52 also includes:

[0138] An extraction unit, used to obtain the mel spectrum of each target voice information, and perform semantic recognition processing on each target voice information to obtain a semantic text corresponding to each target voice information;

[0139] A combining unit, used for combining the mel spectrum and the semantic text corresponding to the same target speech information into speech-text pairs, so as to obtain a plurality of pairs of training samples containing the target environment information;

[0140] A training unit is used to perform training processing on the diffusion model based on the training samples to obtain the target diffusion model used to predict the noise included in the target environment information.

[0141] Furthermore, the training unit is also used for:

[0142] Encoding the mel-spectrograms in the training samples to obtain speech potential representations corresponding to the training samples;

[0143] Obtaining a preset diffusion step, and performing a stepwise noise addition process on the speech potential representation according to the sequence of the diffusion steps to obtain a noisy speech representation;

[0144] Performing text encoding processing on the semantic text in the training sample to obtain semantic encoding features;

[0145] The speech potential representation, the noisy speech representation and the semantic coding features are input into the diffusion model for training processing to obtain the target diffusion model for predicting the noise included in the target environmental information.

[0146] Furthermore, the device further comprises a sampling module, and the sampling module is used for:

[0147] Determine a to-be-processed noisy speech representation from a plurality of noisy speech representations by random sampling; or,

[0148] The multiple noisy speech representations are evaluated to obtain evaluation results corresponding to the respective noisy speech representations; and a noisy speech representation to be processed is determined based on the evaluation results.

[0149] Furthermore, the fusion module 53 is also used for:

[0150] Performing text encoding processing on the semantic text to be processed to obtain semantic encoding features to be processed;

[0151] Performing environment fusion processing on the noisy speech representation to be processed and the semantic coding feature to be processed to obtain intermediate speech information fused with the target environment information;

[0152] Based on the target diffusion model, the intermediate speech information is subjected to step-by-step noise reduction processing in the order of diffusion steps to obtain a fused speech potential representation that incorporates the target environment information;

[0153] Decoding the fused speech latent representation to obtain a fused speech mel spectrum;

[0154] The fused speech mel spectrum is subjected to speech conversion processing to obtain the fused speech information.

[0155] Furthermore, the device also includes a voice library updating module, which is used to:

[0156] receiving user feedback information on the fused voice information;

[0157] Based on the user feedback information, the fused voice information is evaluated and processed to obtain a feedback evaluation result;

[0158] Based on the feedback evaluation result, it is determined whether to update the fused speech information into the speech database.

[0159] The embodiment of the present invention provides a speech generation device that integrates environmental information. Compared with the prior art, the present invention uses target speech information that carries target environmental information to train a diffusion model, and obtains a target diffusion model for predicting noise mixed in the target environmental information, thereby realizing training on a large amount of real audio that carries environmental information, and improving the naturalness and fluency of the integrated speech information. The generation of speech is controlled by two natural language prompts, the target environment description text to be integrated and the semantic text to be processed, without the need for other prompt information. With the control of the target environment description text to be integrated, compared with other speech generation systems in the current financial and medical fields that use text to control the emotion, volume, gender and other factors of the generated speech, the method proposed by the present invention can generate a variety of sounds in different environments, such as speech with sound effects, singing, whispering, etc., further improving the controllability and diversity of the speech generation system.

[0160] According to an embodiment of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction, and the computer executable instruction can execute the speech generation method integrating environmental information in any of the above method embodiments.

[0161] Figure 6 A schematic diagram of the structure of a device provided according to an embodiment of the present invention is shown, and the specific embodiment of the present invention does not limit the specific implementation of the device.

[0162] like Figure 6 As shown, the device may include: a processor (processor) 602 , a communication interface (Communications Interface) 604 , a memory (memory) 606 , and a communication bus 608 .

[0163] The processor 602 , the communication interface 604 , and the memory 606 communicate with each other via a communication bus 608 .

[0164] The communication interface 604 is used to communicate with other devices such as clients or other servers.

[0165] The processor 602 is used to execute the program 610, and specifically can execute the relevant steps of the above-mentioned speech generation method integrating environmental information.

[0166] Specifically, the program 610 may include program codes, which include computer operation instructions.

[0167] The processor 602 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0168] The memory 606 is used to store the program 610. The memory 606 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0169] The program 610 may be specifically configured to enable the processor 602 to perform the following operations:

[0170] Acquire the semantic text to be processed and the target environment description text to be integrated, and determine a plurality of target voice information from the voice library based on the target environment description text to be integrated; the target voice information carries the target environment information corresponding to the target environment description text to be integrated;

[0171] Training a diffusion model based on the plurality of target speech information to obtain a target diffusion model for predicting the noise mixed in the target environmental information;

[0172] Based on the target diffusion model, the semantic text to be processed is subjected to environment fusion processing to obtain fused speech information that incorporates the target environment information.

[0173] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0174] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention. The non-Company software tools or components appearing in the embodiments of the present invention are only examples and do not represent actual use.

Claims

1. A speech generation method integrating environmental information, characterized in that: include: Acquire the semantic text to be processed and the target environment description text to be integrated, and determine a plurality of target voice information from the voice library based on the target environment description text to be integrated; the target voice information carries the target environment information corresponding to the target environment description text to be integrated; Training a diffusion model based on the plurality of target speech information to obtain a target diffusion model for predicting the noise mixed in the target environment information; Based on the target diffusion model, the semantic text to be processed is subjected to environment fusion processing to obtain fused speech information that incorporates the target environment information.

2. The method according to claim 1, characterized in that: The step of determining a plurality of target voice information from a voice library based on the target environment description text to be integrated includes: Performing keyword extraction processing on the description text of the target environment to be integrated into, to obtain keywords of the target environment to be integrated into; Perform keyword matching processing on the target environment keyword to be integrated with the description text of each voice information in the voice library to obtain a keyword matching result; The target voice information is determined based on the keyword matching result.

3. The method according to claim 1, characterized in that: The training process of the diffusion model based on the plurality of target speech information to obtain a target diffusion model for predicting the noise mixed in the target environment information includes: Acquire the mel spectrum of each target voice information, and perform semantic recognition processing on each target voice information to obtain semantic text corresponding to each target voice information; The mel spectrum and the semantic text corresponding to the same target speech information form a speech-text pair to obtain multiple pairs of training samples containing the target environment information; The diffusion model is trained based on the training samples to obtain the target diffusion model used to predict the noise included in the target environment information.

4. The method according to claim 3, characterized in that The training process of the diffusion model based on the training samples to obtain the target diffusion model for predicting the noise included in the target environment information includes: Encoding the mel-spectrograms in the training samples to obtain speech potential representations corresponding to the training samples; Obtaining a preset diffusion step, and performing a stepwise noise addition process on the speech potential representation according to the sequence of the diffusion steps to obtain a noisy speech representation; Performing text encoding processing on the semantic text in the training sample to obtain semantic encoding features; The speech potential representation, the noisy speech representation and the semantic coding features are input into the diffusion model for training processing to obtain the target diffusion model for predicting the noise included in the target environmental information.

5. The method according to claim 1, characterized in that Before performing environment fusion processing on the semantic text to be processed based on the target diffusion model, the method further includes: Determine a to-be-processed noisy speech representation from a plurality of noisy speech representations by random sampling; or, The multiple noisy speech representations are evaluated to obtain evaluation results corresponding to the respective noisy speech representations; and a noisy speech representation to be processed is determined based on the evaluation results.

6. The method according to claim 5, characterized in that The step of performing environment fusion processing on the semantic text to be processed based on the target diffusion model to obtain fused speech information fused with the target environment information includes: Performing text encoding processing on the semantic text to be processed to obtain semantic encoding features to be processed; Performing environment fusion processing on the noisy speech representation to be processed and the semantic coding feature to be processed to obtain intermediate speech information fused with the target environment information; Based on the target diffusion model, the intermediate speech information is subjected to step-by-step noise reduction processing in the order of diffusion steps to obtain a fused speech potential representation that incorporates the target environment information; Decoding the fused speech potential representation to obtain a fused speech mel spectrum; The fused speech mel spectrum is subjected to speech conversion processing to obtain the fused speech information.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: receiving user feedback information on the fused voice information; Based on the user feedback information, the fused voice information is evaluated and processed to obtain a feedback evaluation result; Based on the feedback evaluation result, it is determined whether to update the fused speech information into the speech library.

8. A speech generation device integrating environmental information, characterized in that: include: A target determination module, used to obtain the semantic text to be processed and the target environment description text to be integrated, and determine a plurality of target voice information from the voice library based on the target environment description text to be integrated; the target voice information carries the target environment information corresponding to the target environment description text to be integrated; A training module, used for training a diffusion model based on a plurality of target speech information to obtain a target diffusion model for predicting the noise mixed in the target environment information; The fusion module is used to perform environment fusion processing on the semantic text to be processed based on the target diffusion model to obtain fused speech information that integrates the target environment information.

9. A storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction executes an operation corresponding to the method for generating speech by integrating environmental information as described in any one of claims 1 to 7.

10. A device comprising a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the method for generating speech by integrating environmental information according to any one of claims 1 to 7.

Images