Speech generation method and device fusing environmental information, storage medium, and equipment

Abstract

The application discloses a speech generation method and device fusing environmental information, a storage medium and equipment, relates to the technical field of sound signal processing and the fields of finance and medical treatment, and mainly solves the problem that the speech generated based on the text content in the prior art lacks environmental information in addition to semantic information, and comprises the following steps: obtaining to-be-processed semantic text and to-be-fused target environmental description text, and determining a plurality of target speech information from a speech library based on the to-be-fused target environmental description text; the target speech information carries target environmental information corresponding to the to-be-fused target environmental description text; a diffusion model is trained and processed based on the plurality of target speech information, so that a target diffusion model for predicting noise mixed in the target environmental information is obtained; and the to-be-processed semantic text is subjected to environmental fusion processing based on the target diffusion model, so that fusion speech information fused with the target environmental information is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sound signal processing and the technical field of finance and medicine, and particularly relates to a speech generation method and device fusing environmental information, a storage medium and equipment. BACKGROUND

[0002] In recent years, speech generation systems have made significant progress in the fidelity and diversity of generated speech, gradually improving the clarity and naturalness of synthesized speech. Some speech generation systems have reached a quality level comparable to single-speaker recordings through text prompts. These speech generation systems have demonstrated their powerful ability to generate speech based on semantic information from natural language descriptions and have been widely applied in the technical fields of finance and medicine.

[0003] However, text-prompt-based speech generation systems still face some problems, such as the fact that the textual content of speech is only a part of the information contained in the speech, and the speech generated only through the textual content of the speech lacks information about the environment in which the speech was made and the speaker's style. It is crucial to provide speech generation systems with information other than semantic information, which is related to whether the speech generation system can generate speech that is closer to real human speech. SUMMARY

[0004] Therefore, the present application provides a speech generation method and device fusing environmental information, a storage medium and equipment, which mainly aims to solve the problem that the speech generated based on the textual content of the speech lacks environmental information other than semantic information in the prior art.

[0005] According to one aspect of the present application, a speech generation method fusing environmental information is provided, comprising:

[0006] obtaining a to-be-processed semantic text and a to-be-fused target environment description text, and determining a plurality of target speech information from a speech library based on the to-be-fused target environment description text; the target speech information carries target environmental information corresponding to the to-be-fused target environment description text;

[0007] training and processing a diffusion model based on the plurality of target speech information to obtain a target diffusion model for predicting noise mixed in the target environmental information;

[0008] performing environmental fusion processing on the to-be-processed semantic text based on the target diffusion model to obtain fusion speech information that fuses the target environmental information.

[0009] Further, the determining of the plurality of target speech information from the speech library based on the to-be-fused target environment description text comprises:

[0010] extracting keywords from the target environment description text to be merged, to obtain target environment keywords to be merged;

[0011] matching the target environment keywords to be merged with the description text of each voice information in the voice library to obtain a keyword matching result;

[0012] determining the target voice information based on the keyword matching result.

[0013] Further, the training process of the diffusion model based on the plurality of target voice information to obtain a target diffusion model for predicting noise mixed in the target environment information comprises:

[0014] Obtain the Mel spectrum of each target voice information, and perform semantic recognition processing on each target voice information to obtain semantic text corresponding to each target voice information;

[0015] The Mel spectrum and the semantic text corresponding to the same target voice information form a voice text pair to obtain a plurality of training samples containing the target environment information;

[0016] Training the diffusion model based on the training samples to obtain the target diffusion model for predicting noise mixed in the target environment information.

[0017] Further, the training process of the diffusion model based on the training samples to obtain the target diffusion model for predicting noise mixed in the target environment information comprises:

[0018] Encode the Mel spectrum in the training sample to obtain a voice latent representation corresponding to each training sample;

[0019] Obtain a preset diffusion step and perform step-by-step noise addition processing on the voice latent representation according to the order of the diffusion step to obtain a noisy voice representation;

[0020] Text encoding processing is performed on the semantic text in the training sample to obtain semantic encoding features;

[0021] Input the voice latent representation, the noisy voice representation and the semantic encoding features into the diffusion model for training processing to obtain the target diffusion model for predicting noise mixed in the target environment information.

[0022] Further, before the environment fusion processing of the target semantic text to be processed based on the target diffusion model, the method further comprises:

[0023] determining a to-be-processed noisy speech representation from a plurality of noisy speech representations in a random sampling manner; or

[0024] performing evaluation processing on the plurality of noisy speech representations to obtain evaluation results corresponding to each of the noisy speech representations, and determining a to-be-processed noisy speech representation based on the evaluation results.

[0025] Further, the environment fusion processing of the to-be-processed semantic text based on the target diffusion model to obtain the fusion speech information fused with the target environment information comprises:

[0026] performing text encoding processing on the to-be-processed semantic text to obtain to-be-processed semantic encoding features;

[0027] performing environment fusion processing on the to-be-processed noisy speech representation and the to-be-processed semantic encoding features to obtain intermediate speech information fused with the target environment information;

[0028] performing step-by-step noise reduction processing on the intermediate speech information according to the diffusion order based on the target diffusion model to obtain fusion speech latent representation fused with the target environment information;

[0029] performing decoding processing on the fusion speech latent representation to obtain fusion speech mel-spectrogram;

[0030] performing speech conversion processing on the fusion speech mel-spectrogram to obtain the fusion speech information.

[0031] Further, the method further comprises:

[0032] receiving user feedback information on the fusion speech information;

[0033] performing evaluation processing on the fusion speech information based on the user feedback information to obtain feedback evaluation results;

[0034] determining whether to update the fusion speech information into the speech library based on the feedback evaluation results.

[0035] According to another aspect of the present application, a speech generation device for fusing environment information is provided, comprising:

[0036] a target determination module configured to obtain to-be-processed semantic text and to-be-fused target environment description text, and determine a plurality of target speech information from a speech library based on the to-be-fused target environment description text; the target speech information carries target environment information corresponding to the to-be-fused target environment description text;

[0037] The training module is configured to train the diffusion model based on the plurality of pieces of target speech information, so as to obtain a target diffusion model for predicting noise mixed in the target environment information.

[0038] The fusion module is configured to perform environment fusion processing on the to-be-processed semantic text based on the target diffusion model, so as to obtain fused speech information fused with the target environment information.

[0039] Further, the target determination module is further configured to:

[0040] perform keyword extraction processing on the to-be-fused target environment description text, so as to obtain a to-be-fused target environment keyword;

[0041] perform keyword matching processing on the to-be-fused target environment keyword and description texts of the pieces of speech information in the speech library, so as to obtain a keyword matching result;

[0042] determine the target speech information based on the keyword matching result.

[0043] Further, the training module further comprises:

[0044] The extraction unit is configured to obtain a mel spectrum of each piece of target speech information, and perform semantic recognition processing on each piece of target speech information respectively, so as to obtain a semantic text corresponding to each piece of target speech information;

[0045] The combination unit is configured to combine the mel spectrum and the semantic text corresponding to the same target speech information into a speech text pair, so as to obtain a plurality of pairs of training samples containing the target environment information;

[0046] The training unit is configured to train the diffusion model based on the training samples, so as to obtain the target diffusion model for predicting noise mixed in the target environment information.

[0047] Further, the training unit is further configured to:

[0048] perform encoding processing on the mel spectrum in the training sample, so as to obtain a speech latent representation corresponding to each training sample;

[0049] obtain a preset diffusion step, and perform step-by-step noise adding processing on the speech latent representation according to the sequence of the diffusion steps, so as to obtain a noise-added speech representation;

[0050] perform text encoding processing on the semantic text in the training sample, so as to obtain a semantic encoding feature;

[0051] input the voice latent representation, the noisy voice representation and the semantic coding feature into the diffusion model for training processing to obtain the target diffusion model for predicting noise mixed in the target environmental information.

[0052] Further, the device further comprises a sampling module, which is configured to:

[0053] determine a to-be-processed noisy voice representation from a plurality of noisy voice representations in a random sampling manner; or

[0054] evaluate a plurality of noisy voice representations to obtain an evaluation result corresponding to each of the noisy voice representations, and determine a to-be-processed noisy voice representation based on the evaluation result.

[0055] Further, the fusion module is further configured to:

[0056] perform text coding processing on the to-be-processed semantic text to obtain to-be-processed semantic coding features;

[0057] perform environmental fusion processing on the to-be-processed noisy voice representation and the to-be-processed semantic coding feature to obtain intermediate voice information fused with the target environmental information;

[0058] perform step-by-step noise reduction processing on the intermediate voice information according to the diffusion order of diffusion steps based on the target diffusion model to obtain a fusion voice latent representation fused with the target environmental information;

[0059] perform decoding processing on the fusion voice latent representation to obtain a fusion voice mel-spectrogram;

[0060] perform voice conversion processing on the fusion voice mel-spectrogram to obtain the fusion voice information.

[0061] Further, the device further comprises a voice library updating module, which is configured to:

[0062] receive user feedback information on the fusion voice information;

[0063] perform evaluation processing on the fusion voice information based on the user feedback information to obtain a feedback evaluation result;

[0064] determine whether to update the fusion voice information into the voice library based on the feedback evaluation result.

[0065] According to another aspect of the present application, a storage medium is provided, which stores at least one executable instruction, and the executable instruction causes a processor to perform operations corresponding to the voice generation method for fusing environmental information as described above.

[0066] According to another aspect of the present application, there is provided an apparatus comprising a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface being capable of communicating with each other through the communication bus;

[0067] The memory is configured to store at least one executable instruction, and the executable instruction is configured to enable the processor to perform operations corresponding to the voice generation method fusing environmental information.

[0068] By means of the technical solution described above, the technical solution provided by the embodiment of the present application has at least the following advantages:

[0069] The present application provides a voice generation method and device fusing environmental information, a storage medium and an apparatus. Compared with the prior art, the present application uses target voice information carrying target environmental information to train a diffusion model to obtain a target diffusion model for predicting noise mixed in the target environmental information, realizes training on a large number of real audio carrying environmental information, and improves the naturalness and fluency of the fused voice information. The generation of voice is controlled by two natural language prompts, i.e. a target environmental description text to be fused and a semantic text to be processed, without other prompt information. With the control of the target environmental description text to be fused, compared with other voice generation systems in the current financial and medical fields that use text to control the emotion, volume, gender and other factors of the generated voice, the method proposed in the present application can generate diversified voices in different environments, such as voice with sound effects, singing, whispering, etc., further improving the controllability and diversity of the voice generation system.

[0070] The above description is only a summary of the technical solutions of the present application. In order to enable one skilled in the art to better understand the technical means of the present application, the content of the specification can be implemented, and in order to enable the above and other purposes, features and advantages of the present application to be more apparent and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0071] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not meant to limit the present application. Furthermore, the same reference numerals are used throughout the several views to denote the same or similar parts. In the drawings:

[0072] Figure 1 A flowchart of a voice generation method fusing environmental information provided by an embodiment of the present application is shown;

[0073] Figure 2 A flowchart of another voice generation method fusing environmental information provided by an embodiment of the present application is shown;

[0074] Figure 3 Fig. 1 shows a flow diagram of another method for generating speech by fusing environmental information according to an embodiment of the present application;

[0075] Figure 4 Fig. 2 shows a flow diagram of still another method for generating speech by fusing environmental information according to an embodiment of the present application;

[0076] Figure 5 Fig. 3 shows a structural diagram of a device for generating speech by fusing environmental information according to an embodiment of the present application;

[0077] Figure 6 Fig. 4 shows a structural diagram of a device according to an embodiment of the present application. DETAILED DESCRIPTION

[0078] Exemplary embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings; however, they are not limited to the embodiments set forth herein but can be implemented in various forms. The present embodiments are provided so that this disclosure will be thorough, and will fully convey the scope of the application to those skilled in the art.

[0079] The present embodiments provide a method for generating speech by fusing environmental information, as shown in Fig. 1, the method comprises: Figure 1

[0080] 101, obtaining a semantic text to be processed and a target environmental description text to be fused, and determining a plurality of target speech information from a speech library based on the target environmental description text to be fused; the target speech information carries target environmental information corresponding to the target environmental description text to be fused;

[0081] In the present embodiments, a current execution end obtains a semantic text to be processed and a target environmental description text to be fused. The semantic text to be processed contains semantic information for generating fused speech, and the target environmental description text to be fused represents description information of a fused environment, such as “low noise on the street”, “stream and bird chirping sound in the mountains”, etc., which are not limited in the present embodiments. The current execution end determines a plurality of target speech information from a speech library based on the target environmental description text to be fused. The speech library stores speech information carrying various environmental information, such as speech information for talking on the street, speech information for talking in the valley, speech information for whispering in a low voice, etc., which are not limited in the present embodiments. The target speech information determined in the present embodiments carries target environmental information corresponding to the target environmental description text to be fused. For example, when the target environmental description text to be fused is “low noise on the street”, the speech information for talking on the street is determined as the target speech information, etc., which are not limited in the present embodiments. ​

[0082] 102. training the diffusion model based on the plurality of pieces of target speech information to obtain a target diffusion model for predicting noise mixed in the target environment information;

[0083] In the embodiment of the present application, the current execution end trains the diffusion model based on the plurality of pieces of target speech information to obtain a target diffusion model for predicting noise mixed in the target environment information. The diffusion model (Denoising Diffusion Probabilistic Model, DDPM) includes a forward diffusion process and a reverse diffusion process. In the embodiment of the present application, the forward diffusion process of the diffusion model is layer-by-layer noise adding processing on the target speech information, and the reverse diffusion process is layer-by-layer noise removing processing on the noise-added speech information, thereby restoring the target speech information. Through the forward diffusion process and the reverse diffusion process, the diffusion model can recognize noise mixed in the target environment information, thereby obtaining a target diffusion model for predicting noise mixed in the target environment.

[0084] 103. performing environment fusion processing on the semantic text to be processed based on the target diffusion model to obtain fusion speech information fused with the target environment information.

[0085] In the embodiment of the present application, since the target diffusion model trained in step 102 has the ability to recognize noise mixed in the target environment information, the current execution end can remove noise in the speech information fused with the environment based on the target diffusion model trained in step 102. Specifically, during the environment fusion processing, intermediate speech information mixed with noise and fused with the target environment information needs to be obtained based on the semantic text to be processed, and then the noise in the intermediate speech information fused with the target environment information is processed based on the target diffusion model to obtain fusion speech information fused with the target environment information.

[0086] It should be noted that the method of the present application can be used in human-computer interaction scenes, product promotion scenes, etc. in the fields of finance and medicine, and the embodiment of the present application is not limited.

[0087] Further, as a refinement and extension of the above embodiment, in order to improve the noise recognition ability of the diffusion model, another method for generating speech fused with environment information is provided, as shown in Figure 2 The step determines a plurality of pieces of target speech information from a speech library based on the target environment description text to be fused, including:

[0088] 201. performing keyword extraction processing on the target environment description text to be fused to obtain target environment keywords to be fused;

[0089] In this embodiment of the invention, the current execution terminal extracts keywords from the text describing the target environment to be integrated, thereby obtaining keywords for the target environment to be integrated. For example, when the text describing the target environment to be integrated is "low noise on the street", the keywords "street" and "low noise" can be extracted; when the text describing the target environment to be integrated is "flowing water and birdsong in the mountains", the keywords "mountains", "flowing water" and "birdsong" can be extracted. This embodiment of the invention does not impose specific limitations.

[0090] 202. Perform keyword matching processing on the keywords to be integrated into the target environment and the description text of each voice information in the voice library to obtain keyword matching results;

[0091] In this embodiment of the invention, the current execution terminal performs keyword matching processing on the keywords to be integrated into the target environment obtained in step 201 and the descriptive text of each voice information in the voice library, obtaining keyword matching results of successful or unsuccessful matching. For example, the voice library contains voice information of conversations on the street, conversations in a valley, and whispered conversations, etc., and all of the above voice information has corresponding descriptive text. When performing keyword matching processing on the keywords "street" and "low noise" to be integrated into the target environment in step 201 and the descriptive text of each voice information in the voice library, it can successfully match the descriptive text corresponding to the voice information of conversations on the street in the voice library; it can fail to match the descriptive text corresponding to the voice information of conversations in a valley in the voice library; it can fail to match the descriptive text corresponding to the voice information of whispered conversations in the voice library, etc., and this embodiment of the invention does not make specific limitations.

[0092] 203. Determine the target speech information based on the keyword matching results.

[0093] In this embodiment of the invention, the current execution end determines the successfully matched speech information as the target speech information based on the keyword matching result obtained in step 202, which is then used for the training and processing of the diffusion model in the later stage.

[0094] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, in order to facilitate the training of the diffusion model and improve the efficiency of model training, another speech generation method incorporating environmental information is provided, such as... Figure 3 As shown, the steps involve training the diffusion model based on multiple pieces of target speech information to obtain a target diffusion model for predicting noise mixed in the target environment information, including:

[0095] 301. Obtain the Mel spectrum of each target speech information, and perform semantic recognition processing on each target speech information to obtain the semantic text corresponding to each target speech information;

[0096] In the embodiment of the present application, the current execution end obtains the mel spectrum of each piece of target speech information. The mel spectrum (Mel-spectrogram) integrates the time domain characteristics and frequency domain characteristics of the speech information, and shows the change of the speech frequency with time. Moreover, the current execution end also performs semantic recognition processing on each piece of target speech information to obtain the semantic text corresponding to each piece of target speech information. The semantic recognition processing can be completed by using the automatic speech recognition (ASR) technology, and the embodiment of the present application does not make specific limitation.

[0097] 302, the mel spectrum and the semantic text corresponding to the same target speech information are combined to form a speech text pair, and a plurality of training samples containing the target environmental information are obtained;

[0098] In the embodiment of the present application, the current execution end projects the information of the two modalities of the mel spectrum and the semantic text corresponding to the same target speech information into the same latent space to form a speech text pair, denoted as <speech, text>, thereby obtaining a plurality of training samples containing the target environmental information. When projecting the information of the two modalities of the mel spectrum and the semantic text, the CLAP (Contrastive Language-Audio Pretraining) model can be used for processing. The core of the CLAP model lies in its unique pre-training model, which can extract latent representations from any given audio and text, denoted as c e , which is suitable for various downstream tasks.

[0099] 303, the diffusion model is trained based on the training samples to obtain the target diffusion model for predicting the noise mixed in the target environmental information.

[0100] In the embodiment of the present application, the current execution end trains the diffusion model based on the training samples to obtain the target diffusion model for predicting the noise mixed in the target environmental information. Specifically, the following steps are included:

[0101] (1) The mel spectrum in the training sample is encoded to obtain the speech latent representation corresponding to each training sample, denoted as z0; wherein the encoding can be realized by using the variational autoencoder (VAE), and the embodiment of the present application does not make specific limitation.

[0102] (2) Obtain a preset diffusion step, and perform step-by-step noise adding processing on the speech latent representation according to the order of the diffusion step, to obtain a noise-added speech representation; for example, the above speech latent representation z0 is subjected to first round noise adding processing to obtain a first noise-added speech representation z1; then the first noise-added speech representation is subjected to second round noise adding processing to obtain a second noise-added speech representation z2; then the second noise-added speech representation is subjected to third round noise adding processing to obtain a third noise-added speech representation z3; and so on; then the t-1 noise-added speech representation is subjected to t round noise adding processing to obtain a t noise-added speech representation z t , until all the diffusion steps are completed, to obtain a noise-added speech representation z T after T round noise adding processing. T It should be noted that the noise-added speech representation z t after T round noise adding processing becomes pure Gaussian noise carrying target environmental information.

[0103] (3) The semantic text in the training sample is subjected to text encoding processing to obtain a semantic encoding feature, denoted as c; wherein the text encoding processing is used to convert the semantic text into a high-dimensional vector, and an existing text encoder can be selected to realize the same, and the embodiments of the present application do not make specific limitations.

[0104] (4) The above speech latent representation z0, the noise-added speech representation z t , and the semantic encoding feature c are input into the diffusion model for training processing, to obtain the target diffusion model for predicting the noise mixed in the target environmental information. The target diffusion model obtains z0 by predicting the noise added in z t , and the training target is:

[0105]

[0106] , wherein ε is the noise added in z θ , ε e is the predicted noise, and c e is the latent representation extracted from the audio and the text.

[0107] Further, as a refinement and expansion of the above embodiment, in order to improve the fusion efficiency and fusion effect, another speech generation method for fusing environmental information is provided, and before the step of performing environmental fusion processing on the to-be-processed semantic text based on the target diffusion model, the method further comprises:

[0108] determining a to-be-processed noise-added speech representation from a plurality of noise-added speech representations in a random sampling manner; or

[0109] Multiple noisy speech representations are evaluated to obtain evaluation results corresponding to each noisy speech representation; and a noisy speech representation to be processed is determined based on the evaluation results.

[0110] In this embodiment of the invention, the current execution end can randomly sample from multiple noisy speech representations z. T A noisy speech representation z to be processed is determined. T Alternatively, the current execution end can evaluate and process multiple noisy speech representations to obtain the result of each noisy speech representation z. T The corresponding evaluation results. For example, the ability of the time-of-training diffusion model to reconstruct the latent speech representation z0 is evaluated. After ranking the reconstruction abilities from high to low, the noisy speech representation that is reconstructed best is determined as the noisy speech representation z to be processed. T The embodiments of the present invention are not specifically limited.

[0111] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, in order to obtain fused speech information that incorporates target environmental information more accurately, another speech generation method incorporating environmental information is provided, such as... Figure 4 As shown, the steps involve performing environmental fusion processing on the semantic text to be processed based on the target diffusion model to obtain fused speech information that incorporates the target environmental information, including:

[0112] 401. Perform text encoding processing on the semantic text to be processed to obtain the semantic encoding features to be processed;

[0113] In this embodiment of the invention, the current execution end performs text encoding processing on the semantic text to be processed to obtain the semantic encoded features to be processed, denoted as c'. The text encoding processing is used to convert the semantic text to be processed into a high-dimensional vector, and can be implemented by choosing an existing text encoder; this embodiment of the invention does not impose specific limitations.

[0114] 402. Perform environmental fusion processing on the noisy speech representation to be processed and the semantic coding features to be processed to obtain intermediate speech information that incorporates the target environment information;

[0115] In this embodiment of the invention, the current execution end represents the noisy speech to be processed as z. T The semantic coding feature c obtained in step 401 is subjected to environmental fusion processing to obtain intermediate speech information that incorporates the target environment information; it should be noted that, due to the noisy speech representation z to be processed T The pure Gaussian noise that represents the target environment information is therefore the environmental fusion process in this embodiment of the invention, which is the noise addition process.

[0116] 403、based on the target diffusion model, step-by-step noise reduction processing is performed on the intermediate speech information according to the sequence of diffusion steps, to obtain a fusion speech latent representation that fuses the target environmental information;

[0117] In the embodiment of the application, the current execution end performs step-by-step noise reduction processing on the intermediate speech information obtained in step 402 according to the sequence of diffusion steps based on the target diffusion model trained in steps 301 to 303, that is, performs the inverse diffusion process of the diffusion model, so as to obtain a fusion speech latent representation z0' that fuses the target environmental information.

[0118] 404、decoding processing is performed on the fusion speech latent representation to obtain a fusion speech mel spectrum;

[0119] In the embodiment of the application, the current execution end performs decoding processing on the fusion speech latent representation z0' obtained in step 403 to obtain a fusion speech mel spectrum. The decoding processing process is the inverse operation process of the encoding processing steps recorded in step 303, and in the embodiment of the application, a variational autoencoder (VAE) is preferably used to implement the decoding processing, and the embodiment of the application is not limited in this regard.

[0120] 405、speech conversion processing is performed on the fusion speech mel spectrum to obtain the fusion speech information.

[0121] In the embodiment of the application, the current execution end performs speech conversion processing on the fusion speech mel spectrum obtained in step 404, so as to obtain fusion speech information that fuses the target environmental information. The speech conversion processing can be implemented by using a vocoder, and the embodiment of the application is not limited in this regard.

[0122] Further, as a refinement and expansion of the above embodiment, in order to further improve the effect of environmental fusion, another method for generating speech that fuses environmental information is provided, and the method further comprises:

[0123] receiving user feedback information on the fusion speech information;

[0124] In the embodiment of the application, the current execution end receives user feedback information on the fusion speech information obtained in steps 401 to 405, such as user evaluation or user evaluation score of the fusion speech information, and the embodiment of the application is not limited in this regard.

[0125] based on the user feedback information, performing evaluation processing on the fusion speech information to obtain a feedback evaluation result;

[0126] based on the feedback evaluation result, determining whether to update the fusion speech information to the speech library.

[0127] In the embodiment of the present application, the current execution end evaluates the obtained fusion voice information based on user feedback information to obtain a feedback evaluation result. For example, the user satisfaction degree is evaluated through the user's evaluation to obtain a feedback evaluation result of whether the user is satisfied; or the user's score is compared with a preset score threshold to obtain a feedback evaluation result of whether the score threshold is exceeded, and the like, which is not limited in the embodiment of the present application. Finally, whether the fusion voice information is updated to the voice library is determined based on the feedback evaluation result. For example, the fusion voice information corresponding to the feedback evaluation result of the user's satisfaction is updated to the voice library as a new sample; or the fusion voice information corresponding to the feedback evaluation result of exceeding the score threshold is updated to the voice library as a new sample, and the like, which is not limited in the embodiment of the present application.

[0128] The embodiment of the present application provides a voice generation method fusing environmental information. Compared with the prior art, the present application uses target voice information carrying target environmental information to train a diffusion model to obtain a target diffusion model for predicting noise mixed in the target environmental information, realizes training on a large number of real audio carrying environmental information, and improves the naturalness and fluency of the fusion voice information. The generation of voice is controlled by two natural language prompts, i.e. the target environmental description text to be fused and the semantic text to be processed, without other prompt information. With the control of the target environmental description text to be fused, compared with the current voice generation system in the financial and medical fields which uses text to control the emotion, volume, gender and other factors of the generated voice, the method proposed in the present application can generate diversified voices in different environments, such as voice with sound effects, singing, whispering, etc., further improving the controllability and diversity of the voice generation system.

[0129] As an implementation of the method shown in the above Figure 1 The embodiment of the present application provides a voice generation device fusing environmental information, as shown in the above Figure 5 The device comprises:

[0130] A target determination module 51 is configured to acquire the semantic text to be processed and the target environmental description text to be fused, and determine a plurality of target voice information from a voice library based on the target environmental description text to be fused; the target voice information carries target environmental information corresponding to the target environmental description text to be fused;

[0131] A training module 52 is configured to train a diffusion model based on the plurality of target voice information to obtain a target diffusion model for predicting noise mixed in the target environmental information;

[0132] A fusion module 53 is configured to perform environmental fusion processing on the semantic text to be processed based on the target diffusion model to obtain fusion voice information fusing the target environmental information.

[0133] Further, the target determining module 51 is further configured to:

[0134] perform keyword extraction processing on the target environment description text to be merged to obtain a keyword to be merged into a target environment;

[0135] perform keyword matching processing on the keyword to be merged into a target environment and description text of each voice information in the voice library to obtain a keyword matching result;

[0136] determine the target voice information based on the keyword matching result.

[0137] Further, the training module 52 further comprises:

[0138] an extraction unit configured to obtain a mel spectrum of each piece of the target voice information, and perform semantic recognition processing on each piece of the target voice information to obtain semantic text corresponding to each piece of the target voice information;

[0139] a combination unit configured to combine the mel spectrum and the semantic text corresponding to the same target voice information to form a voice text pair, and obtain multiple pairs of training samples containing the target environment information;

[0140] a training unit configured to train the diffusion model based on the training samples to obtain the target diffusion model for predicting noise mixed in the target environment information.

[0141] Further, the training unit is further configured to:

[0142] perform encoding processing on the mel spectrum in the training sample to obtain a voice latent representation corresponding to each training sample;

[0143] obtain a preset diffusion step, and perform step-by-step noise adding processing on the voice latent representation according to the order of the diffusion step to obtain a noisy voice representation;

[0144] perform text encoding processing on the semantic text in the training sample to obtain a semantic encoding feature;

[0145] input the voice latent representation, the noisy voice representation, and the semantic encoding feature into the diffusion model for training processing to obtain the target diffusion model for predicting noise mixed in the target environment information.

[0146] Further, the apparatus further comprises a sampling module, and the sampling module is configured to:

[0147] determine a to-be-processed noisy voice representation from multiple noisy voice representations in a random sampling manner; or

[0148] The evaluation processing is performed on the plurality of noisy speech representations to obtain evaluation results corresponding to each of the noisy speech representations, and a to-be-processed noisy speech representation is determined based on the evaluation results.

[0149] Further, the fusion module 53 is further used for:

[0150] The to-be-processed semantic text is subjected to text encoding processing to obtain to-be-processed semantic encoding features;

[0151] The to-be-processed noisy speech representation and the to-be-processed semantic encoding features are subjected to environmental fusion processing to obtain intermediate speech information fused with the target environmental information;

[0152] The intermediate speech information is subjected to step-by-step noise reduction processing according to the diffusion order of the target diffusion model to obtain a fused speech latent representation fused with the target environmental information;

[0153] The fused speech latent representation is subjected to decoding processing to obtain a fused speech mel spectrum;

[0154] The fused speech mel spectrum is subjected to speech conversion processing to obtain the fused speech information.

[0155] Further, the device further comprises a speech library updating module, which is used for:

[0156] Receiving user feedback information on the fused speech information;

[0157] Based on the user feedback information, the fused speech information is subjected to evaluation processing to obtain a feedback evaluation result;

[0158] Based on the feedback evaluation result, it is determined whether the fused speech information is updated into the speech library.

[0159] The embodiment of the present application provides a speech generation device fusing environmental information, compared with the prior art, the present application uses target speech information carrying target environmental information to train a diffusion model to obtain a target diffusion model for predicting noise mixed in the target environmental information, realizes training on a large number of real audio carrying environmental information, and improves the naturalness and fluency of the fused speech information. The generation of speech is controlled by two kinds of natural language prompts, i.e., the to-be-fused target environmental description text and the to-be-processed semantic text, without other prompt information. With the control of the to-be-fused target environmental description text, compared with other speech generation systems in the current financial and medical fields which use text to control the generation of speech in terms of emotion, volume, gender and other factors, the method proposed in the present application can generate diversified sounds in different environments, such as speech with sound effects, singing, whispering, etc., further improving the controllability and diversity of the speech generation system.

[0160] According to an embodiment of the present application, a storage medium is provided, and the storage medium stores at least one executable instruction, and the computer executable instruction is used to execute the speech generation method of fusing environment information in any method embodiment.

[0161] Figure 6 A structural diagram of an apparatus according to an embodiment of the present application is shown, and embodiments of the present application do not limit the specific implementation of the apparatus.

[0162] As shown in Figure 6 The apparatus can include a processor 602, a communications interface 604, a memory 606, and a communications bus 608.

[0163] The processor 602, the communications interface 604, and the memory 606 can communicate with each other through the communications bus 608.

[0164] The communications interface 604 is configured to communicate with network elements of other apparatuses, such as clients or other servers.

[0165] The processor 602 is configured to execute the program 610, and specifically can execute the related steps of the speech generation method of fusing environment information.

[0166] Specifically, the program 610 can include program codes, and the program codes include computer operation instructions.

[0167] The processor 602 can be a central processing unit CPU, or an application specific integrated circuit ASIC, or one or more integrated circuits configured to implement embodiments of the present application. The one or more processors included in the apparatus can be processors of the same type, such as one or more CPUs; or can be processors of different types, such as one or more CPUs and one or more ASICs.

[0168] The memory 606 is configured to store the program 610. The memory 606 can include a high-speed RAM memory, and can also include a non-volatile memory, such as at least one disk memory.

[0169] The program 610 can be specifically used to cause the processor 602 to perform the following operations:

[0170] Acquire the to-be-processed semantic text and the to-be-integrated target environment description text, and determine a plurality of target voice information from a voice library based on the to-be-integrated target environment description text; the target voice information carries target environment information corresponding to the to-be-integrated target environment description text;

[0171] Train and process the diffusion model based on the plurality of target voice information, to obtain a target diffusion model for predicting noise mixed in the target environment information;

[0172] Perform environment fusion processing on the to-be-processed semantic text based on the target diffusion model, to obtain fused voice information that has fused the target environment information.

[0173] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present application can be realized by a general computing device, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices, and optionally, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described can be executed in an order different from that shown here, or they can be manufactured into individual integrated circuit modules, or multiple modules or steps thereof can be manufactured into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.

[0174] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application. The non-company software tools or components appearing in the embodiments of the present application are only examples and do not represent actual use.

Claims

1. A speech generation method incorporating environmental information, characterized by, The method comprises the following steps: acquiring a to-be-processed semantic text and a to-be-integrated target environment description text, and determining a plurality of target voice information from a voice library based on the to-be-integrated target environment description text; the target voice information carries target environment information corresponding to the to-be-integrated target environment description text; training a diffusion model based on the plurality of target voice information to obtain a target diffusion model for predicting noise mixed in the target environment information; performing environment fusion processing on the to-be-processed semantic text based on the target diffusion model to obtain fusion voice information fused with the target environment information; the training of the diffusion model based on the plurality of target voice information to obtain the target diffusion model for predicting noise mixed in the target environment information comprises: acquiring a mel spectrum of each piece of target voice information, and performing semantic recognition processing on each piece of target voice information to obtain semantic text corresponding to each piece of target voice information; composing the mel spectrum and the semantic text corresponding to the same target voice information into a voice text pair to obtain a plurality of training samples containing the target environment information; performing encoding processing on the mel spectrum in the training sample to obtain a voice latent representation corresponding to each training sample; acquiring a preset diffusion step, and performing step-by-step noise adding processing on the voice latent representation according to the order of the diffusion steps to obtain a noise-added voice representation; performing text encoding processing on the semantic text in the training sample to obtain semantic encoding features; inputting the voice latent representation, the noise-added voice representation, and the semantic encoding features into the diffusion model for training processing to obtain the target diffusion model for predicting noise mixed in the target environment information.

2. The method of claim 1, wherein, the determination of the plurality of target voice information from the voice library based on the to-be-integrated target environment description text comprises: performing keyword extraction processing on the to-be-integrated target environment description text to obtain to-be-integrated target environment keywords; performing keyword matching processing on the to-be-integrated target environment keywords and the description text of each voice information in the voice library to obtain a keyword matching result; determining the target voice information based on the keyword matching result.

3. The method of claim 1, wherein, Before the environment fusion processing on the to-be-processed semantic text based on the target diffusion model, the method further comprises: determining a to-be-processed noise-added voice representation from a plurality of noise-added voice representations in a random sampling manner; or performing evaluation processing on a plurality of noise-added voice representations to obtain an evaluation result corresponding to each noise-added voice representation; and determining a to-be-processed noise-added voice representation based on the evaluation result.

4. The method of claim 3, wherein, the environment fusion processing on the to-be-processed semantic text based on the target diffusion model to obtain fusion voice information fused with the target environment information comprises: performing text encoding processing on the to-be-processed semantic text to obtain to-be-processed semantic encoding features; performing environment fusion processing on the to-be-processed noise-added voice representation and the to-be-processed semantic encoding features to obtain intermediate voice information fused with the target environment information; based on the target diffusion model, step-by-step noise reduction processing is performed on the intermediate speech information according to the sequence of diffusion steps, to obtain a fusion speech latent representation that fuses the target environmental information; decoding processing is performed on the fusion speech latent representation, to obtain a fusion speech mel spectrum; speech conversion processing is performed on the fusion speech mel spectrum, to obtain the fusion speech information.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: receiving user feedback information on the fusion speech information; based on the user feedback information, evaluation processing is performed on the fusion speech information, to obtain a feedback evaluation result; based on the feedback evaluation result, it is determined whether to update the fusion speech information to the speech library.

6. A speech generating apparatus that fuses environmental information, characterized by comprising: includes: a target determination module configured to obtain a to-be-processed semantic text and a to-be-fused target environment description text, and determine a plurality of target speech information from a speech library based on the to-be-fused target environment description text; the target speech information carries target environmental information corresponding to the to-be-fused target environment description text; a training module configured to train a diffusion model based on the plurality of target speech information, to obtain a target diffusion model for predicting noise mixed in the target environmental information; a fusion module configured to perform environmental fusion processing on the to-be-processed semantic text based on the target diffusion model, to obtain fusion speech information that fuses the target environmental information; The training module further includes: an extraction unit configured to obtain a mel spectrum of each piece of the target speech information, and perform semantic recognition processing on each piece of the target speech information respectively, to obtain a semantic text corresponding to each piece of the target speech information; a combination unit configured to combine the mel spectrum and the semantic text corresponding to the same target speech information to form a speech text pair, to obtain a plurality of training samples containing the target environmental information; a training unit configured to perform encoding processing on the mel spectrum in the training sample, to obtain a speech latent representation corresponding to each training sample; obtain a preset diffusion step, and perform step-by-step noise adding processing on the speech latent representation according to the sequence of the diffusion steps, to obtain a noise-added speech representation; perform text encoding processing on the semantic text in the training sample, to obtain a semantic encoding feature; input the speech latent representation, the noise-added speech representation, and the semantic encoding feature into the diffusion model for training processing, to obtain the target diffusion model for predicting noise mixed in the target environmental information.

7. A storage medium having at least one executable instruction stored therein, the executable instruction performing operations corresponding to the method of generating speech that fuses environmental information according to any one of claims 1-5.

8. A device comprising a processor, a memory, a communication interface, and a communication bus, the processor, the memory, and the communication interface completing communication with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the method of generating speech that fuses environmental information according to any one of claims 1-5.

Images