Method, apparatus, device, medium and program product for generating acoustic features

Abstract

The embodiment of the invention relates to a method, device and equipment for generating acoustic features, a medium and a program product. The method comprises the steps of obtaining a to-be-processed target text and a voice prompt with a target tone; the method further comprises the step of determining text embedding based on the target text and a prompt text corresponding to the voice prompt. The method further comprises the step of determining semantic embedding based on the semantic features of the voice prompt and the semantic features of the target text. The method further includes determining local tone embedding corresponding to a plurality of feature frames of a cue acoustic feature based on the cue acoustic feature corresponding to the voice cue. The method further comprises the step of generating a target acoustic feature which corresponds to the target text and has a target tone based on text embedding, semantic embedding and local tone embedding. According to the method, the input related to the text and the tone of the voice prompt are introduced, so that the generated acoustic features and the voice prompt are more similar, the tone quality and the pronunciation accuracy are improved, and the user experience is improved.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to the field of audio processing, and more specifically to methods, apparatus, devices, media, and program products for generating acoustic features. Background Technology

[0002] Currently, machine learning is playing an increasingly important role in daily life and production. Acoustic models based on deep learning in machine learning have also emerged. Acoustic models have wide applications in speech recognition, speech translation, speech synthesis and other fields. Moreover, acoustic models can not only process related tasks by combining audio, but also combine multimodal content such as text and video to further process corresponding tasks.

[0003] With the development of acoustic models, they can be applied to fields such as speech recognition, speech synthesis, speech conversion, and timbre customization. Therefore, existing acoustic models can perceive a wide variety of sound signals, including speech, audio events, human voices, noise, and timbre. However, many problems still need to be solved in the process of using acoustic models to process audio. Summary of the Invention

[0004] Embodiments of this disclosure provide a method, apparatus, device, medium, and program product for generating acoustic features.

[0005] According to a first aspect of this disclosure, a method for generating acoustic features is provided. The method includes acquiring target text to be processed and a speech prompt having a target timbre. The method further includes determining a text embedding based on the target text and a prompt text corresponding to the speech prompt. The method further includes determining a semantic embedding based on semantic features of the speech prompt and semantic features of the target text. The method further includes determining a local timbre embedding corresponding to multiple feature frames of the prompt acoustic features based on the prompt acoustic features corresponding to the speech prompt. The method further includes generating target acoustic features with a target timbre corresponding to the target text based on the text embedding, semantic embedding, and local timbre embedding.

[0006] In a second aspect of this disclosure, an apparatus for generating acoustic features is provided. The apparatus includes a target text and voice prompt acquisition module configured to acquire target text to be processed and a voice prompt having a target timbre; a text embedding determination module configured to determine a text embedding based on the target text and the prompt text corresponding to the voice prompt; a semantic embedding determination module configured to determine a semantic embedding based on semantic features of the voice prompt and semantic features of the target text; a local timbre embedding determination module configured to determine local timbre embeddings corresponding to multiple feature frames of the prompt acoustic features based on the prompt acoustic features corresponding to the voice prompt; and a target acoustic feature generation module configured to generate target acoustic features having a target timbre corresponding to the target text based on the text embedding, semantic embedding, and local timbre embedding.

[0007] In a third aspect of this disclosure, an electronic device is provided, including at least one processor; and a storage device for storing at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the method according to the first aspect of this disclosure.

[0008] In a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method according to a first aspect of this disclosure.

[0009] In a fifth aspect of this disclosure, a computer program product is provided. This computer program product includes a computer program that, when executed by a processor, implements the method according to a first aspect of this disclosure.

[0010] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0012] Figure 1 The illustration shows a schematic diagram of an example environment in which the devices and / or methods of the embodiments of this disclosure may be implemented;

[0013] Figure 2 The illustration shows a schematic flowchart of an example of generating acoustic features according to an embodiment of the present disclosure;

[0014] Figure 3The illustration shows a schematic diagram of an example method for generating acoustic features according to an embodiment of the present disclosure;

[0015] Figure 4 The illustration shows a schematic diagram of an example of a model for generating acoustic features according to an embodiment of the present disclosure;

[0016] Figure 5 The illustration shows a schematic diagram of another example of a model for generating acoustic features according to an embodiment of the present disclosure;

[0017] Figure 6 The illustration shows a schematic diagram of an example of a diffusion model based on a self-attention mechanism in a model for generating acoustic features according to an embodiment of the present disclosure;

[0018] Figure 7 A schematic block diagram of an apparatus for generating acoustic features according to an embodiment of the present disclosure is shown;

[0019] Figure 8 A schematic block diagram of an example device suitable for implementing embodiments of the present disclosure is shown.

[0020] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0021] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and provisions. Upon receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media performing the operations of this disclosed technical solution, based on the prompt message.

[0022] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0023] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0024] As mentioned above, there are still many problems to be solved in audio generation. For example, in existing voice customization (also known as zero-shot speech synthesis) schemes, the user first provides a voice prompt, and then the model can remember the voice prompt's timbre, pronunciation habits, etc. without training, and can use this timbre for speech synthesis.

[0025] In this scheme, the language model first predicts corresponding coarse-grained semantic features based on the text to be synthesized, such as using Hidden-Unit BERT (HuBERT), BestRQ, or the first layer of the end-to-end neural audio codec SoundStream. Then, the acoustic model converts these coarse-grained semantic features into fine-grained acoustic features, for example, using Mel-Spectrum, hidden layers of Variational Autoencoder (VAE), or later layers of SoundStream. Finally, a vocoder, such as a Mel vocoder, audio VAE, or SoundStream, converts the acoustic features into a speech waveform. However, this scheme suffers from drawbacks when generating fine-grained acoustic features using the acoustic model, including poor sound quality, a lack of similarity between the generated audio and the prompt speech, and insufficient pronunciation accuracy.

[0026] To address at least the aforementioned and other potential problems, embodiments of this disclosure propose a method for generating acoustic features. In this method, a computing device first acquires target text to be processed and a voice prompt with a target timbre. The target text includes corresponding semantic features, and the voice prompt with the target timbre includes corresponding prompt text, semantic features, and prompt acoustic features. Next, the computing device processes the target text and prompt text to obtain text embeddings for the target text and prompt text. The computing device can also process the semantic features of the voice prompt and the semantic features of the target text to obtain corresponding semantic embeddings. Furthermore, the computing device can process the prompt acoustic features corresponding to the voice prompt to obtain local timbre embeddings corresponding to multiple feature frames of the prompt acoustic features. Then, the computing device utilizes the text embedding, semantic embedding, and local timbre embedding to generate target acoustic features with the target timbre corresponding to the target text. Through this method, by introducing the timbre of the text and the voice prompt, the generated acoustic features are more similar to the voice prompt, thereby improving sound quality and pronunciation accuracy, and enhancing the user experience.

[0027] The embodiments of this disclosure will now be described in further detail with reference to the accompanying drawings. Figure 1 An example environment in which the apparatus and / or methods of embodiments of the present disclosure may be implemented is shown. In environment 100, computing device 112 may process a speech prompt 102 with a target timbre and a target text 104 to be processed, and further combine the combined target text and prompt text 106 corresponding to the target text and the speech prompt, semantic features 108, and prompt acoustic features 110 to generate text embeddings 114, semantic embeddings 116, and local timbre embeddings 118, respectively. Finally, computing device generates a target acoustic feature 120 with the target timbre based on text embeddings 114, semantic embeddings 116, and local timbre embeddings 118. The target timbre is an existing timbre in a timbre library or a licensed timbre. Additionally, computing device may also obtain a global timbre embedding associated with the prompt acoustic feature, and the global timbre embedding is also required when generating the target acoustic feature.

[0028] Examples of computing device 112 include, but are not limited to, personal computers, server computers, handheld or laptop devices, mobile devices (such as mobile phones, personal digital assistants (PDAs), media players, etc.), multiprocessor systems, consumer electronics, minicomputers, mainframe computers, and distributed computing environments that include any of the above systems or devices.

[0029] like Figure 1As shown, computing device 112 can be used to receive a voice prompt 102 with a target timbre and target text 104 to be processed. For example, computing device 112 can extract the prompt text corresponding to the voice prompt 102 with the target timbre, and then combine it with the target text 104 to form a combined target text and prompt text 106. For example, the combined target text and prompt text can be generated by concatenating the prompt text and the target text. Thus, computing device 112 can further determine the text embedding 114 of the target text and the prompt text. In one example, the length of text embedding 114 is the sum of the lengths of the target text and the prompt text.

[0030] In some embodiments, when generating the text embedding 114, the computing device 112 obtains the text embedding by applying a combination of the target text and the prompt text corresponding to the voice prompt to a text encoder. The model structure corresponding to the text encoder can be a convolutional neural network with padding or a Transformer structure; this application does not impose any limitations on this.

[0031] The computing device 112 can also extract semantic features corresponding to the speech prompt 102 with the target timbre and semantic features corresponding to the target text 104 to be processed, and further determine semantic features 108 based on the above two semantic features. For example, semantic features 108 are obtained by concatenating the semantic features corresponding to the speech prompt 102 and the semantic features corresponding to the target text 104 to be processed. Then, the computing device 112 can further determine the semantic embedding 116 corresponding to the semantic features 108.

[0032] In some embodiments, the semantic features corresponding to the target text 104 to be processed are obtained by processing a language model. The semantic features of the voice prompt are obtained by processing the voice information of the voice prompt. In some embodiments, when the length of the semantic feature is not equal to the length of the prompt acoustic feature 110 and the acoustic feature to be predicted, the semantic feature 108 can be upsampled or downsampled, or both, to make the length of the semantic feature equal to the length of the prompt acoustic feature 110.

[0033] In some embodiments, when generating the semantic embedding 116, the computing device 112 obtains the semantic embedding by applying the target text and concatenated semantic features corresponding to the voice prompt to a semantic encoder. This is when the lengths of the concatenated semantic features and the acoustic features are not the same, for example, the semantic features are processed at 20 points per second and the acoustic features at 40 points per second. In this case, the semantic encoder can upsample the combined semantic features so that the length of the semantic features is equal to the length of the prompt acoustic feature 110.

[0034] The computing device 112 also extracts cue acoustic features 110 corresponding to the speech cue 102 having the target timbre. Thus, the computing device 112 can further utilize the cue acoustic features 110 to generate a local timbre embedding 118. In one example, the cue acoustic features 110 are Mel-spectral features corresponding to the audio information of a speech cue. In some embodiments, when generating the local timbre embedding 118, the computing device 112 obtains the local timbre embedding by applying the cue acoustic features 110 to a local timbre encoder. The local timbre encoder is a fully connected layer structure such that the feature length of the local timbre embedding is consistent with the length of the combined text and combined semantic features.

[0035] In some embodiments, a corresponding timbre embedding is generated for each frame in the cue acoustic feature 110 using a local timbre encoder. Then, the timbre embeddings corresponding to multiple frames in the cue acoustic feature 110 are combined to form a local timbre embedding 118. Alternatively or additionally, when generating the local timbre embedding 118, initial information of the target acoustic feature corresponding to the target text can be concatenated after the cue acoustic feature 110 to generate a local timbre embedding with the same length as the cue acoustic feature 110 and the target acoustic feature, which is then generated by the local timbre encoder. In one example, the initial information of the target acoustic feature is all 0.

[0036] Finally, the computing device further generates target acoustic features 120 based on text embedding 114, semantic embedding 116, and local timbre embedding 118. In some embodiments, target acoustic features 120 correspond to target text and have a target timbre corresponding to the voice prompt. In some embodiments, the computing device can also generate a global timbre embedding based on the prompt acoustic features 110. For example, the computing device 112 processes the prompt acoustic features 110 as a whole to generate a global timbre embedding, such as using a global timbre encoder. Therefore, target acoustic features can also be generated by text embedding, semantic embedding, local timbre embedding, and global timbre embedding. In addition, the embedding of noisy acoustic features is also required in the process of determining target acoustic features. The above examples are only for describing this disclosure and are not intended to limit the specific scope of this disclosure.

[0037] In some embodiments, the target acoustic features are generated by applying text embeddings, semantic embeddings, local timbre embeddings, and noisy acoustic feature embeddings to a self-attention-based diffusion model. When training the self-attention-based diffusion model, training is performed by acquiring sample text embeddings, sample semantic embeddings, sample local timbre embeddings, sample noisy acoustic features, and sample acoustic features. Additionally, the computing device also requires global timbre embeddings as input. Similarly, when training the self-attention-based diffusion model, the sample global timbre embeddings are further utilized for training.

[0038] This method incorporates text-related input and the timbre of voice prompts into the acoustic model, making the generated acoustic features more similar to the voice prompts, thereby improving sound quality and pronunciation accuracy and enhancing the user experience.

[0039] The above combination Figure 1 The following is a schematic diagram illustrating an example environment in which the devices and / or methods of embodiments of this disclosure may be implemented, in conjunction with... Figure 2 A schematic diagram of a flowchart illustrating an example of generating acoustic features according to embodiments of the present disclosure.

[0040] like Figure 2 As shown, in Example 200, the computing device can be used to acquire target text 202 to be processed and a voice prompt 204 with a target timbre. The computing device can also be used to acquire prompt text 206 for the voice prompt with the target timbre. The computing device can also be used to acquire semantic features 208 of the target text corresponding to the target text to be processed and semantic features 210 of the voice prompt corresponding to the voice prompt. The computing device can also acquire acoustic features 212 of the prompt corresponding to the voice prompt.

[0041] The computing device then combines the target text and the prompt text and applies them to a text encoder to obtain a text embedding 214. The computing device also combines the semantic features 208 of the target text and the semantic features 210 of the voice prompt and applies them to a semantic encoder to obtain a semantic embedding 216. The computing device further applies the prompt acoustic features 212 to a local timbre encoder to obtain a local timbre embedding 218.

[0042] Finally, the computing device 112 can generate target acoustic features 220 based on text embedding 214, semantic embedding 216, and local timbre embedding 218. Additionally, the target acoustic features 220 can be further generated by combining global timbre embedding for the entire speech prompt 204.

[0043] This method incorporates text-related input and the timbre of voice prompts, resulting in generated acoustic features that are more similar to voice prompts, thereby improving sound quality and pronunciation accuracy, and enhancing the user experience.

[0044] The above combination Figure 2 A schematic flowchart illustrating an example of generating acoustic features according to embodiments of the present disclosure is provided below. Figure 3 A schematic diagram illustrating an example method for generating acoustic features according to embodiments of the present disclosure. Figure 3 The process shown can be performed in Figure 1 Executed at the computing device 112 shown or any other suitable computing device.

[0045] like Figure 3 As shown, in Example 300, at box 302, the computing device acquires the target text to be processed and a voice prompt with a target timbre. In some embodiments, the target text to be processed contains a complete sentence, for example, the target text to be processed is "The weather is nice today". Additionally, the target timbre included in the voice prompt is a timbre already in a timbre library or a licensed timbre.

[0046] Subsequently, at box 304, the computing device determines the text embedding based on the target text and the prompt text corresponding to the voice prompt. The target text can be provided to the computing device by the user. The prompt text corresponds to the voice prompt and can be pre-provided text corresponding to the voice prompt, or text generated by the computing device based on the voice prompt.

[0047] In some embodiments, when generating a text embedding, the computing device 112 obtains the text embedding by applying a combination of the target text and the prompt text corresponding to the voice prompt to a text encoder. The text encoder can employ any suitable neural network model, and this application does not impose any limitations on it.

[0048] Next, at box 306, the computing device determines the semantic embedding based on the semantic features of the voice prompt and the semantic features of the target text. The semantic features of the target text are extracted from an existing language model. In one example, when generating the semantic embedding, the computing device obtains the semantic embedding by applying a combination of the semantic features of the target text and the semantic features corresponding to the voice prompt to the semantic encoder.

[0049] In some embodiments, the input to the semantic encoder is a combined semantic feature. Since the semantic embedding and timbre embedding need to be the same size, the length of the combined semantic feature must also be the same as the length of the acoustic feature. When the lengths of the combined semantic feature and the acoustic feature are different, the length of the combined semantic feature can be adjusted to be the same as the length of the acoustic feature, for example, by upsampling or downsampling the combined semantic feature. When the lengths of the combined semantic feature and the acoustic feature are the same, the semantic encoder does not perform a sampling operation.

[0050] Then, at box 308, the computing device determines a local timbre embedding corresponding to multiple feature frames of the prompt acoustic features based on the prompt acoustic features corresponding to the voice prompt. For example, when determining the local timbre embedding, the local timbre encoder generates the local timbre embedding using the prompt acoustic features and the initial information portion of the target acoustic features. Since the length of the initial information portion of the target acoustic features (the initial information portion set to 0) is determined by the target semantic features, the length of the local timbre embedding is determined by the length of the semantic embedding. In some embodiments, the aforementioned multiple feature frames are all feature frames of the prompt acoustic features; for example, the voice prompt is a 60-second audio segment, where 40 feature points can be obtained for each second of audio. When generating the local timbre embedding, the computing device obtains the local timbre embedding by applying multiple feature frames of the prompt acoustic features to the local timbre encoder. Additionally, when generating the local timbre embedding 118, initial information of the target acoustic feature corresponding to the target text can be concatenated after the cue acoustic feature 110, so that the local timbre encoder generates a local timbre embedding corresponding to the lengths of the cue acoustic feature 110 and the target acoustic feature. The initial information of the target acoustic feature is entirely of predetermined values, for example, set to 0.

[0051] In some embodiments, the local timbre encoder is a fully connected layer structure, such that the feature dimension of the local timbre embedding is consistent with the length of the combined text and combined semantic features.

[0052] Finally, at box 310, the computing device generates target acoustic features with the target timbre corresponding to the target text, based on text embedding, semantic embedding, and local timbre embedding. Specifically, the computing device combines the text embedding, semantic embedding, and local timbre embedding to generate a combined embedding. For example, the computing device inputs the combined embedding into a self-attention-based diffusion model, such as a Transformer-based Diffusion model, to generate the target acoustic features.

[0053] In some embodiments, the combined embedding further includes a global timbre embedding, which the computing device obtains by applying cue acoustic features to a global timbre encoder. The global timbre embedding is determined holistically based on the cue acoustic features, and the global timbre encoder outputs a vector. Unlike local timbre embeddings, global timbre embeddings do not have a temporal dimension. In one example, the computing device repeats the global timbre embedding along the temporal dimension to give it the same length as the local acoustic embedding. In another example, the global timbre encoder employs an Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification (ECAPA-TDNN) structure.

[0054] In some embodiments, the combined embedding further includes the embedding of noisy acoustic features. The embedding of noisy acoustic features is obtained by the computing device using a noisy acoustic feature encoder. In one example, the noisy acoustic feature encoder is a fully connected layer structure such that the length of the noisy acoustic feature embedding is the same as the length of the semantic embedding, local timbre embedding, and global timbre embedding.

[0055] In some embodiments, the determination of text embedding, semantic embedding, local timbre embedding, global timbre embedding, and the determination of noisy acoustic feature embedding, as well as the generation of target acoustic features, are achieved through an acoustic model. This acoustic model includes a diffusion model based on a self-attention mechanism to process text embedding, semantic embedding, local timbre embedding, and global timbre embedding to generate the target acoustic features. Additionally, the computing device can acquire sample text embeddings, sample semantic embeddings, sample global timbre embeddings, sample local timbre embeddings, sample noisy acoustic features, and sample acoustic features to train the diffusion model based on the self-attention mechanism, and can further combine sample text and sample speech cues to train the acoustic model.

[0056] In some embodiments, the diffusion model or acoustic model based on the self-attention mechanism can be fine-tuned according to user needs to improve the overall model architecture and adapt to more scenarios and different tasks.

[0057] This method incorporates text-related input and the timbre of voice prompts into the acoustic model, making the generated acoustic features more similar to the voice prompts, thereby improving sound quality and pronunciation accuracy and enhancing the user experience.

[0058] The above combination Figure 3A schematic diagram illustrating an example method for generating acoustic features according to embodiments of the present disclosure is described below. Figure 4 A schematic diagram illustrating an example 400 for generating acoustic features according to embodiments of the present disclosure.

[0059] like Figure 4 As shown, in Example 400, the model architecture consists of a text encoder 404, a semantic encoder 410, a local timbre encoder 418, and a self-attention-based diffusion model 424. It is understood that users can choose appropriate encoders and other types of models and model architectures that can implement the technical solutions in this application according to their own needs; this application does not impose any restrictions.

[0060] When acquiring the target acoustic features corresponding to the target file, a combined text 402 is composed of the prompt text and the target text. The length of the combined text is the sum of the lengths of the prompt text and the target text. The length of the text embedding 406 is the same as the length of the combined text 402. For example, if the length of the prompt text is T1 and the length of the target text is T2, then the length of the generated text embedding 406 is [T1+T2,C], where C represents the length of each column vector. In one example, the model structure corresponding to the text encoder can be a convolutional neural network with padding or a Transformer structure; this application does not impose any restrictions here.

[0061] In some embodiments, the combined semantic feature 408 consists of the semantic features of the voice prompt and the semantic features of the target text. In one example, to make the length of the generated semantic embedding 412 the same as the sum of the lengths of the initial values ​​416 of the prompt acoustic feature 414 and the target acoustic feature, the computing device upsamples or downsamples, or performs both upsampling and downsampling operations on the combined semantic feature 408 through the semantic encoder 410, so that the frequency of the semantic embedding is the same as the frequency of the prompt acoustic feature. For example, the length of the semantic embedding of the voice prompt is T3, which is the same as the length of the acoustic feature of the voice prompt; the length of the semantic embedding of the target text is T4, which is the same as the length of the target acoustic feature. Therefore, the length of the semantic embedding is [T3+T4,C].

[0062] In some embodiments, the cue acoustic feature 414 is combined with the initial value 416 of the target acoustic feature. For example, the initial value 416 of the target acoustic feature is a placeholder representation of all zeros. The local timbre encoder 418 is a fully connected layer structure such that the feature length of the local timbre embedding is consistent with the length of the combined semantic feature, for example, the length is [T3+T4,C].

[0063] The computing device also generates a noisy acoustic feature embedding 434, also of size [T3+T4,C), by inputting the noisy acoustic feature 432 into the noisy acoustic feature encoder 430. During inference, the noisy acoustic feature 432 can be noise targeting the cue acoustic feature and the target acoustic feature. The computing device adds the semantic embedding 412, the local timbre embedding 420, and the noisy acoustic feature embedding 434 to obtain the added feature. The added feature is [T3+T4,C], which is then concatenated with the text embedding 406 to generate a combined embedding 422, of length [T1+T2+T3+T4,C]. The computing device obtains the target acoustic feature 428 by inputting the combined embedding into a self-attention-based diffusion model 424, discarding a portion of the information 426 corresponding to the text embedding and the cue acoustic feature, retaining only the target acoustic feature 428 corresponding to the target text.

[0064] The above process describes the model's inference process. During training, sample speech prompts and their corresponding sample texts are first acquired. The sample texts are then input into a text encoder to obtain sample text embeddings. The semantic features of the sample speech prompts are input into a semantic encoder to obtain sample semantic embeddings. Next, the acoustic features of the sample speech prompts are masked, and the masked sample speech prompts are input into a local timbre encoder to obtain sample local timbre embeddings. Furthermore, the computing device adds noise to the acoustic features of the sample speech prompts to obtain noisy acoustic features, which are then input into a noisy acoustic feature encoder to obtain the embeddings of these noisy acoustic features. The sample semantic embeddings, sample local timbre embeddings, and the embeddings of the noisy acoustic features are then added together, and then concatenated with the sample text embeddings to obtain the sample combined embeddings. The sample combined embeddings are then input into a self-attention-based diffusion model to obtain predicted acoustic features corresponding to the masked acoustic features. These predicted features are then compared with the masked acoustic features to adjust the parameters of the self-attention-based diffusion model in the acoustic model. Similarly, parameters of other parts of the acoustic model can be adjusted during this process.

[0065] The above combination Figure 4 A schematic diagram illustrating an example of a model for generating acoustic features according to embodiments of the present disclosure is shown below. Figure 5 A schematic diagram illustrating Example 500 of another model for generating acoustic features according to embodiments of the present disclosure.

[0066] like Figure 5Example 500, as shown, also includes a combined text 502 generated from the prompt text and the target text, a combined semantic feature 508 generated from the semantic features of the voice prompt and the semantic features of the target text, an acoustic feature 514 for the voice prompt, and a combined acoustic feature 516 consisting of the acoustic features of the voice prompt and the initial value 516 of the target acoustic feature to be predicted. Figure 4 Similarly, the computing device needs to compute text embeddings 506, semantic embeddings 512, and local timbre embeddings 524 corresponding to the combined text, combined semantic features, and local timbre embeddings using text encoder 504, semantic encoder 510, and local timbre encoder 522. Additionally, in example 500, a global timbre embedding 520 needs to be obtained based on acoustic features 514 using a global timbre encoder 518. The obtained global timbre embedding has a size of [1, C], and to ensure that the global timbre embedding and the local timbre embedding have the same size, as shown... Figure 4 As shown, the local timbre embedding has a size of [T3+T4,C], so the global timbre embedding can be repeated to have a size of [T3+T4,C]. Additionally, the computing device inputs the noisy acoustic features 532 into the noisy acoustic feature encoder 530 to obtain the noisy acoustic feature embedding 528.

[0067] Then, the semantic embedding 512, the local timbre embedding 524, the global timbre embedding 520, and the noisy acoustic feature embedding 528 are added together, and then concatenated with the text embedding to form a combined embedding 526. The combined embedding is then input into a self-attention-based diffusion model 534 to generate feature information. The portions 536 of the generated feature information corresponding to the text embedding and the cue acoustic features are discarded, retaining only the target acoustic features 538 corresponding to the target text.

[0068] In some embodiments, the global timbre embedding is determined based on the overall cue acoustic features, and the global timbre encoder outputs a vector. To ensure that the global timbre embedding has the same length as the local timbre embeddings, the global timbre embedding is repeated. For example, the global timbre encoder uses an ECAPA-TDNN structure.

[0069] The above process is mainly a reasoning process, and its training process is similar to... Figure 4Similarly, when generating the global timbre embedding of the samples, the unmasked acoustic features are used and input into the global timbre encoder 518 to obtain the global timbre embedding of the samples. Then, the embeddings of the sample semantic embedding, sample local timbre embedding, sample global timbre embedding, and sample noisy acoustic features are added together, and then concatenated with the text embedding to obtain the sample combined embedding. The sample combined embedding is then input into a self-attention-based diffusion model to obtain the predicted acoustic features, and then compared with the sample acoustic features of the sample cues to adjust the parameters of the self-attention-based diffusion model in the acoustic model. Similarly, the parameters of other parts of the acoustic model can also be further adjusted.

[0070] In some embodiments, the computing device generates a global timbre embedding and a local timbre embedding by masking the sample acoustic features of the sample speech prompt. For example, if the sample speech prompt is "The weather is so nice today!", the computing device can mask one or more of the acoustic features "today", "weather", "nice", and "nice" to set them to 0.

[0071] The above combination Figure 5 A schematic diagram of another example 500 of a model for generating acoustic features according to embodiments of the present disclosure is described below. Figure 6 A schematic diagram illustrating an example of a diffusion model based on a self-attention mechanism in a model for generating acoustic features according to embodiments of the present disclosure.

[0072] like Figure 6 As shown, the self-attention-based diffusion model in the acoustic model consists of multiple Transformer blocks. Input 602 contains a combined embedding, which is an organic combination of text embeddings, semantic embeddings, local timbre embeddings, global timbre embeddings, and noisy acoustic feature embeddings. Output 604 contains the target acoustic features, as well as text-related information and cue acoustic features.

[0073] In some embodiments, the self-attention-based diffusion model includes multiple skip connections, where earlier Transformer blocks are combined with later Transformer blocks through skip connections, thereby improving the execution efficiency of the self-attention-based diffusion model.

[0074] Figure 7 A schematic block diagram of an apparatus for generating acoustic features according to an embodiment of the present disclosure is shown. Figure 7As shown, the device 700 includes a target text and voice prompt acquisition module 710, configured to acquire the target text to be processed and a voice prompt with a target timbre; a text embedding determination module 720, configured to determine text embedding based on the target text and the prompt text corresponding to the voice prompt; a semantic embedding determination module 730, configured to determine semantic embedding based on the semantic features of the voice prompt and the semantic features of the target text; a local timbre embedding determination module 740, configured to determine local timbre embedding corresponding to multiple feature frames of the prompt acoustic features based on the prompt acoustic features corresponding to the voice prompt; and a target acoustic feature generation module 750, configured to generate target acoustic features with a target timbre corresponding to the target text based on text embedding, semantic embedding, and local timbre embedding.

[0075] In some embodiments, the text embedding determination module 720 includes: a combined text determination module configured to obtain combined text by combining target text and prompt text; and a text embedding determination module configured to obtain the text embedding based on the combined text.

[0076] In some embodiments, the text embedding determination module includes a text encoder application module configured to obtain text embedding by applying combined text to the text encoder.

[0077] In some embodiments, the semantic embedding determination module 730 includes: a semantic feature generation module for voice prompts, configured to generate semantic features for voice prompts based on the audio of the voice prompts; a semantic feature generation module for target text, configured to generate semantic features for target text based on the text in the target text; a combined semantic feature generation module, configured to generate combined semantic features by combining the semantic features of the voice prompts and the semantic features of the target text; and a semantic embedding generation module, configured to generate a semantic embedding based on the combined semantic features.

[0078] In some embodiments, the semantic embedding generation module includes a semantic encoder application module configured to generate semantic embeddings by applying combined semantic features to the semantic encoder.

[0079] In some embodiments, the local timbre embedding determination module 740 includes: a feature frame determination module configured to determine a plurality of feature frames in the cue acoustic features; and a local timbre encoder application module configured to determine local timbre embedding for the cue acoustic features by applying the plurality of feature frames to a local timbre encoder.

[0080] In some embodiments, the target acoustic feature generation module 750 includes: a global timbre embedding determination module configured to determine a global timbre embedding for the cue acoustic features based on the overall cue acoustic features; and a target acoustic feature generation module configured to generate target acoustic features with target timbre corresponding to the target text based on text embedding, semantic embedding, local timbre embedding and global timbre embedding.

[0081] In some embodiments, the target acoustic feature generation module 750 further includes: a noisy acoustic feature embedding acquisition module configured to acquire the embedding of noisy acoustic features corresponding to the cue acoustic features and the target acoustic features; a combined embedding generation module configured to generate a combined embedding by combining text embedding, semantic embedding, local timbre embedding, global timbre embedding and the embedding of noisy acoustic features; and a target acoustic feature generation module configured to generate target acoustic features with target timbre corresponding to the target text based on the combined embedding.

[0082] In some embodiments, the combined embedding generation module further includes: a local timbre embedding length determination module configured to determine the length of the local timbre embedding; a global timbre embedding adjustment module configured to adjust the global timbre embedding based on the length of the local timbre embedding; and a combined embedding generation module configured to generate a combined embedding based on text embedding, semantic embedding, local timbre embedding, adjusted global timbre embedding, and embedding of noisy acoustic features.

[0083] In some embodiments, the global timbre embedding adjustment module further includes a global timbre embedding repetition module, configured to generate an adjusted global timbre embedding by repeating the global timbre embedding based on the length of the local timbre embedding.

[0084] In some embodiments, the combined embedding generation module further includes a diffusion model application module based on a self-attention mechanism, configured to generate target acoustic features with target timbre corresponding to the target text by applying the combined embedding to a diffusion model based on a self-attention mechanism.

[0085] In some embodiments, the apparatus 700 further includes: a sample text and sample speech prompt acquisition module, configured to acquire sample text and sample speech prompts having sample timbres and corresponding to the sample text; a sample text embedding determination module, configured to determine sample text embeddings based on the sample text; a sample semantic embedding determination module, configured to determine sample semantic embeddings based on the sample speech prompts; a sample global timbre embedding and sample local timbre embedding generation module, configured to generate sample global timbre embeddings and sample local timbre embeddings by masking the sample acoustic features of the sample speech prompts; and a diffusion model training module based on a self-attention mechanism, configured to train a diffusion model based on a self-attention mechanism based on sample text embeddings, sample semantic embeddings, sample global timbre embeddings, sample local timbre embeddings, sample noisy acoustic features targeting sample acoustic features, and sample acoustic features.

[0086] Figure 8 A schematic block diagram of an example device 800 that can be used to implement embodiments of the present disclosure is shown. Figure 1 The computing device 112 can be implemented using device 800. As shown, device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 802 or loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 can also store various programs and data required for the operation of device 800. CPU 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0087] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage page 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0088] The various processes and handling described above, such as method 300, can be executed by processing unit 801. For example, in some embodiments, method 300 can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU 801, one or more actions of method 300 described above can be performed.

[0089] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0090] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0091] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0092] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0093] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0094] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0095] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0097] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating acoustic features, comprising: Obtain the target text to be processed and the voice prompt with the target timbre; Based on the target text and the prompt text corresponding to the voice prompt, determine the text embedding; Based on the semantic features of the voice prompt and the semantic features of the target text, the semantic embedding is determined; Based on the acoustic features of the prompt corresponding to the voice prompt, local timbre embeddings corresponding to multiple feature frames of the prompt acoustic features are determined; as well as Based on the text embedding, the semantic embedding, and the local timbre embedding, target acoustic features corresponding to the target text and possessing the target timbre are generated.

2. The method according to claim 1, wherein determining the text embedding based on the target text and the prompt text corresponding to the voice prompt includes: Combined text is obtained by combining the target text and the prompt text; as well as The text embedding is obtained based on the combined text.

3. The method of claim 2, wherein obtaining the text embedding based on the combined text comprises: The text embedding is obtained by applying the combined text to a text encoder.

4. The method according to claim 1, wherein determining the semantic embedding based on the semantic features of the voice prompt and the semantic features of the target text includes: Based on the audio of the voice prompt, semantic features for the voice prompt are generated; Based on the text in the target text, generate semantic features for the target text; Combined semantic features are generated by combining the semantic features of the voice prompt and the semantic features of the target text; as well as The semantic embedding is generated based on the combined semantic features.

5. The method according to claim 4, wherein generating the semantic embedding based on the combined semantic features comprises: The semantic embedding is generated by applying the combined semantic features to the semantic encoder.

6. The method according to claim 1, wherein determining the local timbre embedding corresponding to a plurality of feature frames of the prompt acoustic features based on the prompt acoustic features corresponding to the voice prompt includes: Identify multiple feature frames in the acoustic features of the prompt; as well as The local timbre embedding for the cue acoustic features is determined by applying the plurality of feature frames to a local timbre encoder.

7. The method according to claim 1, wherein generating target acoustic features corresponding to the target text and having the target timbre based on the text embedding, the semantic embedding, and the local timbre embedding comprises: Based on the overall acoustic features of the prompt, the global timbre embedding for the acoustic features of the prompt is determined; as well as Based on the text embedding, the semantic embedding, the local timbre embedding, and the global timbre embedding, target acoustic features corresponding to the target text and possessing the target timbre are generated.

8. The method according to claim 7, wherein generating target acoustic features corresponding to the target text and having the target timbre based on the text embedding, the semantic embedding, the local timbre embedding, and the global timbre embedding comprises: Obtain the embedding of noisy acoustic features corresponding to the prompt acoustic features and the target acoustic features; A combined embedding is generated by combining the text embedding, the semantic embedding, the local timbre embedding, the global timbre embedding, and the embedding of the noisy acoustic features; and Based on the combined embedding, target acoustic features with the target timbre corresponding to the target text are generated.

9. The method of claim 8, wherein generating the combined embedding by combining the text embedding, the semantic embedding, the local timbre embedding, the global timbre embedding, and the embedding of the noisy acoustic features comprises: Determine the length of the local timbre embedding; The global timbre embedding is adjusted based on the length of the local timbre embedding; as well as A combined embedding is generated based on the text embedding, the semantic embedding, the local timbre embedding, the adjusted global timbre embedding, and the embedding of the noisy acoustic features.

10. The method of claim 9, wherein adjusting the global timbre embedding based on the length of the local timbre embedding comprises: Based on the length of the local timbre embedding, the adjusted global timbre embedding is generated by repeating the global timbre embedding.

11. The method of claim 9, wherein generating the combined embedding based on the text embedding, the semantic embedding, the local timbre embedding, the adjusted global timbre embedding, and the embedding of the noisy acoustic features comprises: By applying the combined embedding to a diffusion model based on a self-attention mechanism, target acoustic features with the target timbre corresponding to the target text are generated.

12. The method of claim 11, wherein training the diffusion model based on the self-attention mechanism comprises: Obtain sample text and sample voice prompts with sample timbre that correspond to the sample text; Based on the sample text, determine the sample text embedding; Based on the sample voice prompts, determine the sample semantic embedding; The acoustic features of the sample speech prompts are masked to generate global and local timbre embeddings; and The diffusion model based on the self-attention mechanism is trained based on the sample text embedding, the sample semantic embedding, the sample global timbre embedding, the sample local timbre embedding, the sample noisy acoustic features for the sample acoustic features, and the sample acoustic features.

13. An apparatus for generating acoustic features, comprising: The target text and voice prompt acquisition module is configured to acquire the target text to be processed and the voice prompt with the target timbre; The text embedding determination module is configured to determine the text embedding based on the target text and the prompt text corresponding to the voice prompt; The semantic embedding determination module is configured to determine the semantic embedding based on the semantic features of the voice prompt and the semantic features of the target text; The local timbre embedding determination module is configured to determine the local timbre embedding corresponding to multiple feature frames of the prompt acoustic features based on the prompt acoustic features corresponding to the voice prompt. as well as The target acoustic feature generation module is configured to generate target acoustic features corresponding to the target text and having the target timbre based on the text embedding, the semantic embedding, and the local timbre embedding.

14. An electronic device comprising: At least one processor; as well as A storage device for storing at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the method according to any one of claims 1-10.

15. A computer-readable storage medium having a computer program stored thereon, the computer program implementing the method according to any one of claims 1-12 when executed by a processor.

16. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-12.

Images