Audio generation model processing method, audio generation method, apparatus, and electronic device

Abstract

The present disclosure relates to an audio generation model processing method, an audio generation method, an apparatus and an electronic device. The model processing method comprises: obtaining instruction text, content text corresponding to a sample audio generation task, and target audio; the sample audio generation task comprises at least two task types; inputting the instruction text, the content text and the target audio into a to-be-trained audio generation model to generate instruction encoding, content encoding and target audio encoding; splicing the instruction encoding and the content encoding to generate text encoding, and using a noise sample to obtain a noisy audio encoding according to a time parameter; generating a predicted vector field based on the task text encoding, the time parameter and the noisy audio encoding, and constructing a real vector field based on the noise sample and the target audio encoding; and training the audio generation model according to the difference between the predicted vector field and the real vector field. The audio generation model trained by the present disclosure can improve the diversity of the task types of the executable audio generation tasks.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to an audio generation model processing method, an audio generation method, an apparatus, and an electronic device. Background Technology

[0002] With the development of artificial intelligence technology, a technology has emerged that uses audio generation models to convert text into audio. This technology can include generating understandable and natural speech waveforms based on text content using audio generation models, i.e., text-to-speech; generating music audio based on text descriptions using audio generation models, i.e., text-to-music; and generating sound effects based on open text descriptions using audio generation models, i.e., text-to-sound effects.

[0003] In related technologies, text-to-speech, text-to-music, and text-to-sound effects are accomplished using different audio generation models, resulting in a fragmented modality. Currently, there is a lack of an audio generation model that can simultaneously handle text-to-speech, text-to-music, and text-to-sound effects tasks, thus existing audio generation models perform relatively limited audio generation tasks. Summary of the Invention

[0004] This disclosure provides an audio generation model processing method, an audio generation method, an apparatus, and an electronic device, to at least address the problem that audio generation models in related technologies perform relatively limited types of audio generation tasks. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, an audio generation model processing method is provided, comprising:

[0006] Obtain the sample task instruction text, sample task content text, and target audio that matches the sample audio generation task corresponding to the sample audio generation task; the sample audio generation task includes at least two task types;

[0007] The sample task instruction text, the sample task content text, and the target audio are input into the audio generation model to be trained to generate the instruction code corresponding to the sample task instruction text, the content code corresponding to the sample task content text, and the target audio code corresponding to the target audio.

[0008] The instruction encoding and the content encoding are concatenated to generate the task text encoding corresponding to the sample audio generation task. Then, according to the time parameters associated with the sample audio generation task, the target audio encoding is noise-added using noise samples to obtain noise-added audio encoding.

[0009] Based on the task text encoding, the time parameters, and the noise-added audio encoding, a prediction vector field is generated, and based on the difference between the noise samples and the target audio encoding, a true vector field is constructed.

[0010] The audio generation model is trained based on the difference between the predicted vector field and the true vector field to obtain the trained audio generation model.

[0011] In an exemplary embodiment, the content encoding is generated based on the phoneme sequence of the sample task content text. When the sample audio generation task is a text-to-sound effect task, the phoneme sequence of the sample task content text is obtained through the following steps: obtaining a preset global scaling factor; the global scaling factor is used to characterize the ratio between the phoneme sequence length and the audio duration; obtaining the phoneme sequence length corresponding to the sample task content text according to the audio duration of the target audio and the global scaling factor; generating repeating preset symbol units according to the phoneme sequence length corresponding to the sample task content text, and using the repeating preset symbol units as the phoneme sequence of the sample task content text.

[0012] In an exemplary embodiment, the global scaling factor is obtained through the following steps: obtaining the phoneme sequence length corresponding to each preset speech sample and the audio duration corresponding to each preset speech sample; obtaining the ratio between the phoneme sequence length and the audio duration corresponding to each preset speech sample based on the phoneme sequence length and the audio duration; and using the average of the ratios as the global scaling factor.

[0013] In an exemplary embodiment, the training of the audio generation model includes multiple training stages; obtaining the sample task instruction text, sample task content text, and target audio matching the sample audio generation task corresponding to the sample audio generation task includes: obtaining the sample audio generation task matching the current training stage, and the sample task instruction text, sample task content text, and target audio matching the sample audio generation task corresponding to the sample audio generation task; inputting the sample task instruction text, sample task content text, and target audio into the audio generation model to be trained includes: inputting the sample task instruction text, sample task content text, and target audio into the audio generation model matching the current training stage; after training the audio generation model, the method further includes: if the current training stage is not the last of the multiple training stages, taking the next training stage of the current training stage as the new current training stage, and taking the audio generation model as the new audio generation model matching the current training stage, returning to the step of obtaining the sample audio generation task matching the current training stage, until the current training stage is the last of the multiple training stages, and taking the audio generation model as the trained audio generation model.

[0014] In one exemplary embodiment, the sample audio generation task includes a text-to-speech task, a text-to-music task, and a text-to-sound effects task. The multiple training stages include a first training stage, a second training stage, and a third training stage executed sequentially. Obtaining a sample audio generation task matching the current training stage includes: when the current training stage is the first training stage, the sample audio generation task matching the current training stage is a first sample audio generation task; the first sample audio generation task includes the text-to-speech task; when the current training stage is the second training stage, the sample audio generation task matching the current training stage is a second sample audio generation task; the second sample audio generation task includes the text-to-speech task and the text-to-music task; when the current training stage is the third training stage, the sample audio generation task matching the current training stage is a third sample audio generation task; the third sample audio generation task includes the text-to-speech task, the text-to-music task, and the text-to-sound effects task.

[0015] In an exemplary embodiment, generating a prediction vector field based on the task text encoding, the time parameter, and the noise-added frequency encoding includes: inputting the task text encoding, the time parameter, and the noise-added frequency encoding into a joint diffusion layer of the audio generation model; performing joint attention processing on the task text encoding and the noise-added frequency encoding using the time parameter to obtain an audio encoding aligned with the task text encoding; inputting the audio encoding aligned with the time parameter and the task text encoding into a single-path diffusion layer of the audio generation model; performing self-attention refinement processing on the audio encoding aligned with the task text encoding using the time parameter to obtain a refined audio encoding; and inputting the refined audio encoding into the output layer of the audio generation model to obtain the prediction vector field.

[0016] According to a second aspect of the present disclosure, an audio generation method is provided, comprising:

[0017] Obtain the task instruction text corresponding to the audio generation task to be processed, and the task content text corresponding to the audio generation task to be processed;

[0018] The task instruction text, task content text, and preset noise signal are input into the trained audio generation model to generate the instruction code corresponding to the task instruction text, the content code corresponding to the task content text, and the noise audio code corresponding to the preset noise signal; the audio generation model is trained by the audio generation model training method as described in any embodiment of the first aspect.

[0019] The instruction encoding and the content encoding are concatenated to generate the task text encoding corresponding to the audio generation task to be processed. Based on the task text encoding, the preset time parameters and the noise audio encoding, the prediction vector field corresponding to the audio generation task to be processed is generated.

[0020] The predicted vector field is numerically integrated to obtain the target audio code, and the target audio code is decoded to obtain the target audio corresponding to the audio generation task to be processed.

[0021] According to a third aspect of the present disclosure, an audio generation model processing apparatus is provided, comprising:

[0022] The training sample acquisition unit is configured to acquire the sample task instruction text, sample task content text, and target audio matching the sample audio generation task corresponding to the sample audio generation task; the sample audio generation task includes at least two task types.

[0023] The sample encoding acquisition unit is configured to input the sample task instruction text, the sample task content text, and the target audio into the audio generation model to be trained, and generate the instruction encoding corresponding to the sample task instruction text, the content encoding corresponding to the sample task content text, and the target audio encoding corresponding to the target audio.

[0024] The sample encoding processing unit is configured to concatenate the instruction encoding with the content encoding to generate the task text encoding corresponding to the sample audio generation task, and to add noise to the target audio encoding using noise samples according to the time parameters associated with the sample audio generation task to obtain noise-added audio encoding.

[0025] The vector field construction unit is configured to generate a predicted vector field based on the task text encoding, the time parameters, and the noise-added audio encoding, and to construct a true vector field based on the difference between the noise samples and the target audio encoding.

[0026] The generative model training unit is configured to train the audio generation model based on the difference between the predicted vector field and the true vector field to obtain the trained audio generation model.

[0027] In an exemplary embodiment, the content encoding is generated based on the phoneme sequence of the sample task content text. When the sample audio generation task is a text-to-sound effect task, the training sample acquisition unit is further configured to acquire a preset global scaling factor. The global scaling factor characterizes the ratio between the phoneme sequence length and the audio duration. Based on the audio duration of the target audio and the global scaling factor, the phoneme sequence length corresponding to the sample task content text is acquired. Repeated preset symbol units are generated according to the phoneme sequence length corresponding to the sample task content text, and the repeated preset symbol units are used as the phoneme sequence of the sample task content text.

[0028] In an exemplary embodiment, the training sample acquisition unit is further configured to acquire the phoneme sequence length corresponding to each preset speech sample and the audio duration corresponding to each preset speech sample; acquire the ratio between the phoneme sequence length and the audio duration corresponding to each preset speech sample based on the phoneme sequence length and the audio duration; and use the average of the ratios as the global scaling factor.

[0029] In one exemplary embodiment, the training of the audio generation model includes multiple training phases; the training sample acquisition unit is further configured to acquire sample audio generation tasks matching the current training phase, as well as sample task instruction text and sample task content text corresponding to the sample audio generation tasks, and target audio matching the sample audio generation tasks; the sample encoding acquisition unit is further configured to input the sample task instruction text, the sample task content text, and the target audio into the audio generation model matching the current training phase; the generation model training unit is further configured to, if the current training phase is not the last of the multiple training phases, take the next training phase of the current training phase as the new current training phase, and take the audio generation model as the new audio generation model matching the current training phase, and return to the step of acquiring sample audio generation tasks matching the current training phase, until the current training phase is the last of the multiple training phases, and take the audio generation model as the trained audio generation model.

[0030] In one exemplary embodiment, the sample audio generation task includes a text-to-speech task, a text-to-music task, and a text-to-sound effects task. The plurality of training stages include a first training stage, a second training stage, and a third training stage executed sequentially. The training sample acquisition unit is further configured to, when the current training stage is the first training stage, have the sample audio generation task matched by the current training stage as a first sample audio generation task; the first sample audio generation task includes the text-to-speech task; when the current training stage is the second training stage, have the sample audio generation task matched by the current training stage as a second sample audio generation task; the second sample audio generation task includes the text-to-speech task and the text-to-music task; when the current training stage is the third training stage, have the sample audio generation task matched by the current training stage as a third sample audio generation task; the third sample audio generation task includes the text-to-speech task, the text-to-music task, and the text-to-sound effects task.

[0031] In one exemplary embodiment, the vector field construction unit is further configured to perform a joint diffusion layer of the audio generation model, inputting the task text encoding, the time parameter, and the noise-added frequency encoding into the audio generation model; performing joint attention processing on the task text encoding and the noise-added frequency encoding using the time parameter to obtain an audio encoding aligned with the task text encoding; inputting the audio encoding aligned with the time parameter and the task text encoding into a single-path diffusion layer of the audio generation model; performing self-attention refinement processing on the audio encoding aligned with the task text encoding using the time parameter to obtain a refined audio encoding; and inputting the refined audio encoding into the output layer of the audio generation model to obtain the prediction vector field.

[0032] According to a fourth aspect of the present disclosure, an audio generation apparatus is provided, comprising:

[0033] The task information acquisition unit is configured to acquire the task instruction text corresponding to the audio generation task to be processed, and the task content text corresponding to the audio generation task to be processed.

[0034] The task encoding acquisition unit is configured to execute an audio generation model trained by inputting the task instruction text, the task content text, and a preset noise signal, and generate an instruction encoding corresponding to the task instruction text, a content encoding corresponding to the task content text, and a noise audio encoding corresponding to the preset noise signal; the audio generation model is trained by the audio generation model training method as described in any embodiment of the first aspect.

[0035] The vector field prediction unit is configured to concatenate the instruction encoding and the content encoding to generate the task text encoding corresponding to the audio generation task to be processed, and generate the prediction vector field corresponding to the audio generation task to be processed based on the task text encoding, the preset time parameters and the noise audio encoding.

[0036] The target audio generation unit is configured to perform numerical integration on the prediction vector field to obtain the target audio encoding, and to decode the target audio encoding to obtain the target audio corresponding to the audio generation task to be processed.

[0037] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement an audio generation model processing method as described in any embodiment of the first aspect, or an audio generation method as described in any embodiment of the second aspect.

[0038] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform an audio generation model processing method as described in any embodiment of the first aspect, or an audio generation method as described in any embodiment of the second aspect.

[0039] According to a seventh aspect of the present disclosure, a computer program product is provided, the computer program product including instructions that, when executed by a processor of an electronic device, enable the electronic device to perform an audio generation model processing method as described in any embodiment of the first aspect, or an audio generation method as described in any embodiment of the second aspect.

[0040] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0041] The process involves acquiring sample task instruction text, sample task content text, and target audio matching the sample audio generation task. The sample audio generation task includes at least two task types. The sample task instruction text, sample task content text, and target audio are input into the audio generation model to be trained, generating instruction codes corresponding to the sample task instruction text, content codes corresponding to the sample task content text, and target audio codes corresponding to the target audio. The instruction codes and content codes are concatenated to generate the task text code corresponding to the sample audio generation task. Noise is added to the audio code using noise samples according to the time parameters associated with the sample audio generation task, resulting in a noise-added frequency code. A prediction vector field is generated based on the task text code, time parameters, and the noise-added frequency code. A true vector field is constructed based on the difference between the noise samples and the target audio code. The audio generation model is trained based on the difference between the prediction vector field and the true vector field to obtain the trained audio generation model. This application can collect instruction text, content text, and target audio corresponding to various types of sample audio generation tasks. The task instruction text, task content text, and target audio are then input into the audio generation model to be trained to generate instruction codes corresponding to the task instruction text, content codes corresponding to the task content text, and target audio codes corresponding to the target audio. The instruction codes and content codes are then concatenated to generate task text codes. Furthermore, noise is added to the audio codes using time parameters and noise samples to obtain noise-added frequency codes. These task text codes, time parameters, and noise-added frequency codes are then combined to generate a prediction vector field for vector field learning and model training. Audio generation models trained in this way can perform sample audio generation tasks of various task types, thus improving the diversity of executable audio generation tasks.

[0042] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0044] Figure 1 This is a flowchart illustrating an audio generation model processing method according to an exemplary embodiment.

[0045] Figure 2 This is a flowchart illustrating the acquisition of a phoneme sequence of sample task content text according to an exemplary embodiment.

[0046] Figure 3 This is a flowchart illustrating a task for generating sample audio that matches the training phase, according to an exemplary embodiment.

[0047] Figure 4 This is a flowchart illustrating an audio generation method according to an exemplary embodiment.

[0048] Figure 5 This is a structural diagram illustrating an audio generation model based on conditional flow matching according to an exemplary embodiment.

[0049] Figure 6 This is a block diagram illustrating an audio generation model processing apparatus according to an exemplary embodiment.

[0050] Figure 7 This is a block diagram illustrating an audio generation apparatus according to an exemplary embodiment.

[0051] Figure 8 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0052] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0053] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0054] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0055] Figure 1 This is a flowchart illustrating an audio generation model processing method according to an exemplary embodiment, such as... Figure 1 As shown, the audio generation model processing method is used in an audio generation device and includes the following steps.

[0056] In step S101, the sample task instruction text, sample task content text, and target audio matching the sample audio generation task are obtained. The sample audio generation task includes at least two task types.

[0057] A sample audio generation task refers to the audio generation task used to train an audio generation model. This task may include task instruction text, which describes the style, emotion, accent, and other multi-dimensional attributes of the generated audio, as well as task content text, which describes the content of the generated audio. The sample audio generation task can also be associated with a target audio, which refers to audio that satisfies the sample audio generation task. Furthermore, the sample audio generation task used to train the model can include various task types, such as text-to-speech, text-to-music, and text-to-sound effects tasks. Therefore, the trained audio generation model can perform various types of audio generation tasks.

[0058] Specifically, the audio generation device can collect multiple sample audio generation tasks of different types, and can obtain the sample task instruction text, sample task content text, and target audio that matches each sample audio generation task.

[0059] In step S102, the sample task instruction text, sample task content text, and target audio are input into the audio generation model to be trained to generate the instruction code corresponding to the sample task instruction text, the content code corresponding to the sample task content text, and the target audio code corresponding to the target audio.

[0060] The audio generation model to be trained refers to the artificial intelligence model that needs to be trained to complete the audio generation task. The instruction encoding can refer to the embedding sequence corresponding to the sample task instruction text, the content encoding refers to the embedding sequence corresponding to the sample task content text, and the target audio encoding refers to the audio latent variables corresponding to the target audio.

[0061] Specifically, after obtaining the sample task instruction text, sample task content text, and target audio, the audio generation device can use this information as model input to train the audio generation model. This audio generation model can carry an instruction encoding module, such as a frozen large language model, a pre-trained text encoder, or a lightweight sentence vector model, which can be used to encode the sample task instruction text and generate instruction codes. Simultaneously, the audio generation model can also carry a content encoding module. This content encoding module can first convert the sample task content text into a phoneme sequence through a character-to-phoneme conversion method, and then map it into content codes using a phoneme sequence encoder, such as Zipformer, Conformer, Transformer, or convolutional networks. Furthermore, the audio generation model can also include an audio encoder, such as a Mel-VAE encoder or other acoustic representation autoencoders, such as EnCodec or DAC, to generate the target audio code corresponding to the target audio.

[0062] In step S103, the instruction encoding and content encoding are concatenated to generate the task text encoding corresponding to the sample audio generation task. Then, according to the time parameters associated with the sample audio generation task, the target audio encoding is noise-added using noise samples to obtain the noise-added audio encoding.

[0063] Task text encoding refers to the encoding information of task text related to the sample audio generation task. This task text encoding can be obtained by concatenating instruction encoding and content encoding. For example, the instruction encoding and content encoding can be concatenated in the sequence dimension. Then, the time parameters associated with the task can be generated according to the sample audio, thereby using noise samples to add noise to the target audio encoding to obtain noise-added audio encoding.

[0064] For example, time parameters Satisfy At this point, add noise frequency encoding It can be obtained through interpolation and can be represented as:

[0065]

[0066] in, Indicates a noise sample. Indicates the target audio encoding.

[0067] In step S104, a prediction vector field is generated based on the task text encoding, time parameters, and noise-added frequency encoding, and a real vector field is constructed based on the difference between the noise samples and the target audio encoding.

[0068] In step S105, an audio generation model is trained based on the difference between the predicted vector field and the real vector field to obtain the trained audio generation model.

[0069] Vector fields can be used to characterize how noise samples change to target audio codes. In this embodiment, the audio generation model can construct a predicted vector field based on the task text code, time parameters, and noise-added frequency codes. At the same time, it can construct a true vector field based on the difference between the noise samples and the target audio codes. Thus, the model loss value is obtained by using the difference between the predicted vector field and the true vector field, and the training of the audio generation model is completed using the model loss value.

[0070] For example, the loss function can be as follows:

[0071]

[0072] in, The predicted vector field can be represented based on time parameters. Task text encoding With noise-enhanced frequency coding get, It can represent a real vector field, where This represents a noise sample, while This indicates the target audio encoding, while This indicates that the average value is taken.

[0073] In the above-mentioned audio generation model processing method, the following steps are taken: First, the sample task instruction text, sample task content text, and target audio matching the sample audio generation task are obtained. The sample audio generation task includes at least two task types. The sample task instruction text, sample task content text, and target audio are input into the audio generation model to be trained, generating instruction codes corresponding to the sample task instruction text, content codes corresponding to the sample task content text, and target audio codes corresponding to the target audio. The instruction codes and content codes are concatenated to generate the task text code corresponding to the sample audio generation task. Noise is added to the audio code using noise samples according to the time parameters associated with the sample audio generation task, resulting in a noise-added frequency code. Based on the task text code, time parameters, and noise-added frequency code, a prediction vector field is generated. Based on the difference between the noise samples and the target audio code, a true vector field is constructed. The audio generation model is trained based on the difference between the prediction vector field and the true vector field to obtain the trained audio generation model. This application can collect instruction text, content text, and target audio corresponding to various types of sample audio generation tasks. The task instruction text, task content text, and target audio are then input into the audio generation model to be trained to generate instruction codes corresponding to the task instruction text, content codes corresponding to the task content text, and target audio codes corresponding to the target audio. The instruction codes and content codes are then concatenated to generate task text codes. Furthermore, noise is added to the audio codes using time parameters and noise samples to obtain noise-added frequency codes. These task text codes, time parameters, and noise-added frequency codes are then combined to generate a prediction vector field for vector field learning and model training. Audio generation models trained in this way can perform sample audio generation tasks of various task types, thus improving the diversity of executable audio generation tasks.

[0074] In one exemplary embodiment, content encoding is generated based on the phoneme sequence of the sample task content text. In the case where the sample audio generation task is a text-to-sound effects task, such as... Figure 2 As shown, the phoneme sequence of the sample task content text is obtained through the following steps:

[0075] In step S201, a preset global scaling factor is obtained; the global scaling factor is used to characterize the ratio between the phoneme sequence length and the audio duration.

[0076] In this embodiment, content encoding can be generated based on the phoneme sequence of the sample task content text. For example, the sample task content text can first be converted into a phoneme sequence using a character-to-phoneme conversion method, and then mapped to content encoding using a phoneme sequence encoder. However, this process can only be used for certain types of sample audio generation tasks, such as text-to-speech and text-to-music tasks. The sample task content text for text-to-speech tasks can be speech content text, while the sample task content text for text-to-music tasks can be lyrics text. Therefore, both can be converted into phoneme sequences using a character-to-phoneme conversion method. For text-to-sound effects tasks, there is no transcribed sample task content text; the corresponding sample task content text is usually blank text. Therefore, other methods are needed to generate the phoneme sequence to generate the corresponding content encoding.

[0077] Specifically, if the sample audio generation task is a text-to-sound effect task, the audio generation device can first obtain a pre-set global scaling factor, which can characterize the ratio between the length of the phoneme sequence and the duration of the audio.

[0078] In step S202, the length of the phoneme sequence corresponding to the sample task content text is obtained based on the audio duration of the target audio and the global scaling factor.

[0079] In step S203, repeating preset symbol units are generated according to the length of the phoneme sequence corresponding to the sample task content text, and the repeating preset symbol units are used as the phoneme sequence of the sample task content text.

[0080] The preset symbol unit can refer to a pre-set, learnable special symbol unit, i.e., a pseudo-phoneme unit, which can be denoted as [SFX]. In this embodiment, for the text-to-sound effect task, a phoneme sequence can be constructed by repeating the preset symbol unit. The length of the phoneme sequence, i.e. the number of times the preset symbol unit is repeated, can be determined according to the audio duration of the target audio.

[0081] Specifically, after obtaining the global scaling factor that represents the ratio between the length of the phoneme sequence and the duration of the audio, the audio generation device can determine the length of the corresponding phoneme sequence by combining the duration of the target audio. Then, based on the length of the phoneme sequence, it can determine the number of repetitions of the preset symbol unit, thereby constructing a repeating preset symbol unit that matches the number of repetitions, and using it as the phoneme sequence of the sample task content text.

[0082] For example, the generation of phoneme sequences can be described by the following expression:

[0083]

[0084] in, This indicates the generation of a phoneme sequence. If the preset symbol unit is represented, then It represents the length of the phoneme sequence and can characterize the number of repetitions of a preset symbol unit. This indicates a repeated splicing operation.

[0085] The length of the phoneme sequence is calculated using the following expression:

[0086]

[0087] in, Indicates the length of the phoneme sequence. Represents the global scaling factor, then This indicates the duration of the target audio. This represents the floor function.

[0088] In this embodiment, content encoding can be generated based on the phoneme sequence of the sample task content text. However, for the text-to-sound effect task, since there is a lack of phoneme sequences that can be directly converted from text, the content text can be generated by using repeated preset symbol units as phoneme sequences. The number of repetitions of the preset symbol units, i.e. the length of the phoneme sequence, can be obtained based on the audio duration of the target audio. In this way, by injecting preset symbol units, unstructured sound effects can be mapped into content sequences that can be processed isomorphically with phonemes, thereby ensuring that the audio generation model accurately processes the text-to-sound effect task.

[0089] Furthermore, the global scaling factor is obtained through the following steps: obtaining the phoneme sequence length and audio duration corresponding to each preset speech sample; obtaining the ratio between the phoneme sequence length and audio duration corresponding to each preset speech sample based on the phoneme sequence length and audio duration; and using the average of these ratios as the global scaling factor.

[0090] A preset speech sample can refer to an audio sample carrying speech content. The phoneme sequence length corresponding to the preset speech sample refers to the sequence length of the phoneme sequence converted from the speech content text containing the preset speech sample. The audio duration corresponding to the preset speech sample refers to the duration of the preset speech sample. In this embodiment, the audio generation device can pre-collect multiple speech samples and count the phoneme sequence length and audio duration of each speech sample.

[0091] Then, the audio generation device can calculate the ratio between the phoneme sequence length and the audio duration of each speech sample, and use the average of the ratios as a global scaling factor.

[0092] For example, the global scaling factor can be calculated using the following formula:

[0093]

[0094] in, Indicates the global scaling factor. Indicates the first The length of the phoneme sequence of a speech sample. express The audio duration of each speech sample This represents the total number of speech samples.

[0095] In this embodiment, the ratio between the phoneme sequence length and the audio duration of each preset speech sample can be calculated based on the phoneme sequence length and audio duration corresponding to the preset speech sample, and the average value of the ratio can be used as the global scaling factor. This method can improve the accuracy of obtaining the global scaling factor.

[0096] In an exemplary embodiment, the training of the audio generation model includes multiple training stages; step S101 may further include: acquiring a sample audio generation task matching the current training stage, and the target audio matching the sample audio generation task with the sample task instruction text and sample task content text corresponding to the sample audio generation task; step S102 may further include: inputting the sample task instruction text, sample task content text, and target audio into the audio generation model matching the current training stage; step S105 may further include: if the current training stage is not the last of multiple training stages, taking the next training stage of the current training stage as the new current training stage, and taking the audio generation model as the new audio generation model matching the current training stage, returning to execute step S101, until the current training stage is the last of multiple training stages, and taking the audio generation model as the trained audio generation model.

[0097] In this embodiment, the audio generation model can be trained by multiple training stages. The current training stage refers to any one of the multiple training stages. Specifically, the audio generation device can first determine the current training stage of the audio generation model, and then select matching sample audio generation tasks according to the current training stage, and then obtain the sample task instruction text, sample task content text and target audio corresponding to the sample audio generation task.

[0098] Then, the sample task instruction text, sample task content text, and target audio can be input into the audio generation model matched for the current training stage, i.e., the audio generation model that needs to be trained in the current training stage. The audio generation model generates a prediction vector field, and the difference between the prediction vector field and the real vector field is used to complete the training of the current training stage. Whether the current training stage is completed can be controlled according to the training round threshold. That is, after the training round reaches the training round threshold, it is determined that the current training stage is completed, and it is determined whether the current training stage is the last of multiple training stages.

[0099] If it is not the last training phase, then the next training phase after the current training phase will be taken as the new current training phase, and the audio generation model that has completed the training of the current training phase will be taken as the audio generation model that matches the new current training phase. The sample audio generation task that matches the new current training phase will be selected again, and the training will be completed again until the current training phase is the last training phase. Then the audio generation model that has completed the training of the current training phase will be taken as your trained audio generation model, and the training process will end.

[0100] In this embodiment, the audio generation model can be trained in multiple stages, and the sample audio generation task used can be selected according to each training stage. This multi-stage training method can effectively solve problems such as optimization conflicts, destruction of speech details, or difficulty in convergence that may be caused by joint training of multiple types of sample audio generation tasks, thereby improving the stability of audio generation model training.

[0101] Furthermore, the sample audio generation task includes text-to-speech, text-to-music, and text-to-sound effects tasks, with multiple training phases including a first training phase, a second training phase, and a third training phase executed sequentially; such as Figure 3 As shown, the task of generating sample audio that matches the current training phase can further include:

[0102] In step S301, if the current training stage is the first training stage, the sample audio generation task matched in the current training stage is the first sample audio generation task; the first sample audio generation task includes text-to-speech task.

[0103] The various types of sample audio generation tasks can be text-to-speech, text-to-music, and text-to-sound effects tasks. The training phase of the audio generation model can also be divided into three sequentially executed phases: the first training phase, the second training phase, and the third training phase.

[0104] If the current training phase is the first training phase, that is, the first training phase, the audio generation device can use the first sample audio generation task as the sample audio generation task matched in the current training phase. The first sample audio generation task can refer to the text-to-speech task. That is, in the first training phase, only the speech dataset is used to complete the model training, which can enable the model to first establish a stable instruction following and phoneme alignment capability and obtain robust stream matching dynamics.

[0105] In step S302, if the current training stage is the second training stage, the sample audio generation task matched in the current training stage is the second sample audio generation task; the second sample audio generation task includes text-to-speech task and text-to-music task.

[0106] If the current training phase is the second training phase, the audio generation device can use the second sample audio generation task as the sample audio generation task matched for the current training phase. The second sample audio generation task can include text-to-speech tasks and text-to-music tasks. That is, in the second training phase, music data can be introduced after the first training phase is completed and mixed with speech for training, so that the model can expand the musical semantics such as style, orchestration, and rhythm on the basis of the established alignment mechanism.

[0107] In step S303, if the current training stage is the third training stage, the sample audio generation task matched in the current training stage is the third sample audio generation task; the third sample audio generation task includes text-to-speech task, text-to-music task and text-to-sound effects task.

[0108] If the current training phase is the third training phase, i.e. the last training phase, the audio generation device can use the third sample audio generation task as the sample audio generation task matched for the current training phase. The third sample audio generation task can simultaneously include text-to-speech, text-to-music, and text-to-sound effects tasks. That is, in the training of the third phase, sound effect data can be introduced after the completion of the second training phase and mixed with speech and music for training to complete the joint modeling of non-linguistic audio.

[0109] In this embodiment, the sample audio generation task includes text-to-speech, text-to-music, and text-to-sound effects tasks. The multiple training stages include a first training stage, a second training stage, and a third training stage executed sequentially. The sample audio generation task used in the first training stage is the text-to-speech task, the sample audio generation task used in the second training stage is the text-to-speech task + text-to-music task, and the sample audio generation task used in the third training stage is the text-to-speech task + text-to-music task + text-to-sound effects task. This approach allows the audio generation model to simultaneously complete the text-to-speech, text-to-music, and text-to-sound effects tasks, and avoids problems such as optimization conflicts, destruction of speech details, or convergence difficulties, thereby improving the stability of the audio generation model in processing multiple types of audio generation tasks.

[0110] In an exemplary embodiment, step S104 may further include: inputting the task text encoding, time parameters, and noise-added frequency encoding into the joint diffusion layer of the audio generation model; performing joint attention processing on the task text encoding and noise-added frequency encoding using the time parameters to obtain an audio encoding aligned with the task text encoding; inputting the audio encoding aligned with the time parameters and the task text encoding into the single-path diffusion layer of the audio generation model; performing self-attention refinement processing on the audio encoding aligned with the task text encoding using the time parameters to obtain a refined audio encoding; and inputting the refined audio encoding into the output layer of the audio generation model to obtain a prediction vector field.

[0111] In this embodiment, the audio generation model can include multiple joint diffusion layers, namely joint diffusion Transformer layers. These joint diffusion layers are mainly used to align the audio encoding with the task text encoding. Specifically, the audio generation device can input the task text encoding, time parameters, and noise-added frequency encoding into the joint diffusion layer of the audio generation model. This joint diffusion layer can use the time parameters to perform joint attention processing on the task text encoding and the noise-added frequency encoding. For example, the task text encoding and the noise-added frequency encoding can first undergo self-attention to model intra-modal dependencies, and then joint attention can be performed. The query, key, and value of the task text encoding and the noise-added frequency encoding are concatenated in the attention calculation, so that the audio side can simultaneously focus on the global semantics of the instruction and the fine-grained positional information of the content sequence, thereby achieving the alignment of the audio encoding with the task text encoding, that is, obtaining the audio encoding aligned with the task text encoding.

[0112] Simultaneously, the audio generation model can also include multiple single-path diffusion layers, namely single-path diffusion Transformer layers. These single-path diffusion layers are mainly used for detailed modeling of the audio encoding, completing the refinement process. Specifically, after obtaining the audio encoding aligned with the task text encoding through the joint diffusion layer, the audio generation device can input the audio encoding and time parameters into the single-path diffusion layer of the audio generation model. This single-path diffusion layer can use the time parameters to perform self-attention refinement processing on the audio encoding, thereby obtaining a refined audio encoding.

[0113] Finally, the audio generation device can input the refined audio code into the output layer of the audio generation model, thereby mapping the refined audio code into the final prediction vector field.

[0114] In this embodiment, a joint diffusion layer and a single-path diffusion layer can also be set in the audio generation model. First, the audio encoding and the task text encoding are aligned through the joint diffusion layer, and then the audio encoding is refined through the single-path diffusion layer to finally generate a prediction vector field. This method can improve the accuracy of the prediction vector field generation.

[0115] Figure 4 This is a flowchart illustrating an audio generation method according to an exemplary embodiment, such as... Figure 4 As shown, the audio generation method is used in an audio generation device and includes the following steps.

[0116] In step S401, the task instruction text corresponding to the audio generation task to be processed and the task content text corresponding to the audio generation task to be processed are obtained.

[0117] The audio generation task to be processed refers to an audio generation task that requires processing and generates audio from text. The task instruction text refers to the task instruction text input for the audio generation task to be processed, while the task content text refers to the task content text input for the audio generation task to be processed. Specifically, the audio generation device can receive both the task instruction text and the task content text input for the audio generation task to be processed.

[0118] In step S402, the task instruction text, task content text, and preset noise signal are input into the trained audio generation model to generate the instruction code corresponding to the task instruction text, the content code corresponding to the task content text, and the noise audio code corresponding to the preset noise signal; the audio generation model is trained by the audio generation model training method described in any of the above embodiments.

[0119] The preset noise signal can be a pre-set Gaussian noise signal. After obtaining the task instruction text and task content text, the task instruction text, task content text and preset noise signal can be input into the trained audio generation model. The audio generation model generates the instruction code corresponding to the task instruction text, the content code corresponding to the task content text, and the noise audio code corresponding to the preset noise signal.

[0120] For example, it could be an instruction encoding module that takes task instruction text as input and outputs the corresponding instruction encoding, or a content encoding module that takes task content text as input and first converts the task content text into a phoneme sequence, then maps it to content encoding. It could also perform inference on a preset noise signal to obtain noise audio encoding.

[0121] In step S403, the instruction encoding and content encoding are concatenated to generate the task text encoding corresponding to the audio generation task to be processed. Based on the task text encoding, the pre-set time parameters, and the noise audio encoding, the prediction vector field corresponding to the audio generation task to be processed is generated.

[0122] Then, the instruction encoding and content encoding can be concatenated to generate the task text encoding corresponding to the audio generation task to be processed. Based on this task text encoding, pre-set time parameters, and noise audio encoding, a prediction vector field corresponding to the audio generation task to be processed can be generated. For example, the task text encoding, pre-set time parameters, and noise audio encoding can be input into the joint diffusion layer and single-path diffusion layer of the audio generation model to generate the prediction vector field.

[0123] In step S404, the predicted vector field is numerically integrated to obtain the target audio code, and the target audio code is decoded to obtain the target audio corresponding to the audio generation task to be processed.

[0124] The target audio refers to the audio signal ultimately generated by the audio generation task to be processed, and the target audio code refers to the audio code corresponding to the audio signal. After obtaining the prediction vector field, the audio generation device can perform numerical integration on the prediction vector field, which can be ODE numerical integration, such as Euler method or Runge-Kutta method, to obtain the target audio code. Finally, it is decoded by a Mel spectrum decoder to obtain the final target audio.

[0125] In the aforementioned audio generation method, the task instruction text and task content text corresponding to the audio generation task to be processed are obtained; the task instruction text, task content text, and a preset noise signal are input into a trained audio generation model to generate instruction codes corresponding to the task instruction text, content codes corresponding to the task content text, and noise audio codes corresponding to the preset noise signal; the audio generation model is trained using the audio generation model training method described above; the instruction codes and content codes are concatenated to generate task text codes corresponding to the audio generation task to be processed, and a prediction vector field corresponding to the audio generation task to be processed is generated based on the task text codes, preset time parameters, and noise audio codes; the prediction vector field is numerically integrated to obtain the target audio code, and the target audio code is decoded to obtain the target audio corresponding to the audio generation task to be processed. This disclosure allows the input of the task content text corresponding to the audio generation task to be processed into a trained audio generation model to output a prediction vector field, thereby obtaining the final target audio based on the prediction vector field. Since the trained audio generation model can perform sample audio generation tasks of various task types, this model can be used to complete various types of audio generation tasks to be processed, thus improving the task type diversity of the processed audio generation tasks.

[0126] In one exemplary embodiment, a unified audio generation method based on text instructions is also provided. This method can achieve unified audio generation based on conditional flow matching (CFM). It employs a dual-channel multimodal diffusion transformer (MM-DiT) as its backbone, decoupling the "natural language instruction" and "content sequence" into two types of conditional flows, and performing temporal concatenation and fusion on the text conditional flow side. On the audio side, the continuous latent variables output by a Mel-spectral variational autoencoder (Mel-VAE) are used as the generation object. The vector field is learned through flow matching, and the clean latent variables are recovered through numerical integration of ordinary differential equations, followed by decoding to obtain the waveform.

[0127] The structure of the audio generation model can be as follows: Figure 5 As shown, it may include the following modules:

[0128] (a) Instruction encoding module (frozen large language model or equivalent semantic encoder):

[0129] Function: Encodes natural language instruction text into instruction embedding sequences. The dimension example is batch size. Instruction length Hidden Dimensions .

[0130] Typical input differences:

[0131] Speech synthesis: Instructions describe speaker attributes (gender, age, emotion, style, accent, etc.). In multi-speaker dialogue scenarios, speaker identifier prefix rules can be used to distinguish different speaker segments.

[0132] Music generation: Instructions describe the style, instrumentation, tempo, mood, and whether vocals are included.

[0133] Sound effect generation: Instructions describe acoustic events or scenes (such as barking dogs in the street, distant thunder, etc.).

[0134] (ii) Content Encoding Module (Structured Content Related to the Task):

[0135] Function: Generate content embedding sequences The length is denoted as .

[0136] For speech and music: The text or lyrics are converted into a phoneme sequence via grapheme-to-phoneme (G2P) conversion, and then mapped by a content encoding network (such as a Zipformer-based encoder or other sequence encoders) to... This provides fine-grained time structure conditions that can be aligned with acoustic processes.

[0137] For sound effects: There is no transcribed phoneme sequence. Because sound effects lack discrete phoneme / lyric content, forcibly using a single global duration embedding makes it difficult to form progressively unfolding time anchors within the attention mechanism. Completely avoiding the serialized content path would be inconsistent with the conditional organization of speech / music, disrupting the unified architecture. Therefore, this embodiment uses learnable special symbols (e.g., denoted as [SFX]) to repeatedly construct the content sequence, with the number of repetitions determined by the target duration. Specifically, it may include the following steps:

[0138] 1. Introduce learnable special symbol units (pseudophone units) into the vocabulary, denoted as [SFX] (the name is only an example and can be replaced with any fixed special token).

[0139] 2. Calculate the average ratio of phoneme sequence length to audio duration on the speech corpus to obtain the global scaling factor. :

[0140]

[0141] in Indicates the first The length of the phoneme sequence of a speech sample. express The audio duration of each speech sample This represents the total number of speech samples.

[0142] 3. For the target sound effect generation request, specify the target duration. Construct a content sequence:

[0143]

[0144]

[0145] That is, the number of times a special symbol is repeated increases linearly with the target duration.

[0146] 4. Obtained through content encoding .

[0147] (III) Condition Fusion:

[0148] Concatenate the instruction embedding and content embedding in the sequence dimension:

[0149] , shape is .

[0150] This concatenation is used as a text modality condition in subsequent joint attention layers, enabling the model to simultaneously obtain global style / semantic control (mainly from instructions) and temporal structure / expansion control (mainly from content sequences).

[0151] (iv) Audio latent variable paths and Mel-VAE:

[0152] The original audio (e.g., 44.1kHz sampling rate) is compressed into continuous latent variables using a pre-trained Mel-VAE. Latent space has a higher degree of temporal abstraction compared to waveforms.

[0153] During the training phase, interpolation states are constructed within the stream matching framework:

[0154] right ,Pick (in (Can be noise samples), and learn vector fields. .

[0155] (V) MM-DiT Dual-Path Structure and Interaction Method (Main Improvements: Joint Attention + Audio Refinement):

[0156] The text path (conditional path) and the audio path (generation path) are stacked in parallel:

[0157] 1. In several "joint diffusion Transformer layers" (the number is denoted as...) (in Chinese) text conditions With audio latent variables First, each component performs self-attention to model intra-modal dependencies. Then, joint attention is performed: the queries, keys, and values ​​of the two paths are concatenated in the attention calculation, enabling the audio side to simultaneously perceive the global semantics of the instructions and the fine-grained positional information of the content sequence, thereby achieving alignment of "style determined by instructions, structure determined by content".

[0158] 2. After the joint layer, the audio latent variables enter several "single-path diffusion Transformer layers" (the number of which is denoted as...). (This involves performing self-attention refinement only on the audio to improve the ability to model acoustic texture details.)

[0159] Position coding can employ relative position coding such as RoPE to enhance timing awareness.

[0160] (vi) Training objective (conditional flow matching loss):

[0161] The optimization objective is to estimate the vector field so that the predicted velocity approximates the true displacement.

[0162]

[0163] Among them, the expected value Data samples and conditional sampling.

[0164] This objective is the core mathematical constraint of the technical solution, and the specific norm can be L2 or other differentiable measures.

[0165] Furthermore, speech requires strong alignment and clear pronunciation. Music requires long-term structure and rhythmic stability. Sound effect texture variance is large. If full-modal mixed training is started from scratch, optimization conflicts, destruction of speech details, or convergence difficulties are likely to occur. Therefore, this embodiment adopts a three-stage course strategy, and the algorithm logic is as follows:

[0166] Phase 1 (Speech Anchoring): The model is trained using only a speech dataset to first establish a stable ability to follow instructions and align phonemes, and to obtain robust stream matching dynamics.

[0167] Phase Two (Semantic Expansion: Speech + Music): After Phase One is completed, music data is introduced and trained in combination with speech data, so that the model can expand the musical semantics such as style, orchestration, and rhythm on the basis of the established alignment mechanism.

[0168] Phase 3 (Full Modal Generalization: Speech + Music + Sound Effects): After Phase 2 is completed, sound effect data is introduced, and joint modeling of non-linguistic audio is completed under the dynamic special symbol injection mechanism.

[0169] Phase switching can be controlled by training round thresholds: for example, set E1 to be the number of rounds in phase one and E2 to be the number of extra rounds in phase two. When the number of training rounds exceeds the threshold, switch the data mixing set.

[0170] Each training step still executes the same CFM loss and the same input process: constructing instruction sequences and content sequences (phoneme sequences or [SFX] sequences) based on the batch sample type, and sampling. calculate With the regression vector field.

[0171] (vii) Reasoning process:

[0172] Given instructions and content conditions, starting from noise initialization, the learned vector field is numerically integrated using ODE (e.g., Euler method or other higher-order solvers) to obtain clean latent variables, which are then decoded into Mel spectrum / waveform by Mel-VAE.

[0173] The specific task processing is as follows:

[0174] 1. Text-to-speech task:

[0175] Input: Instruction text describing timbre and emotion. Content: Phoneme sequence of the reading transcript via G2P.

[0176] System execution: Qwen2.5-7B output instruction embedding sequence Zipformer output content embedding sequence After concatenation, the MM-DiT engine generates Mel-VAE latent variables and decodes them into speech.

[0177] 2. Text-to-Music Task:

[0178] Input: Commands describing the musical style / instrumentation / emotion. Content: Lyrics G2P phoneme sequence.

[0179] System execution: Same as text-to-speech task, but the instruction semantic space and music control labels are generated according to a unified template during the data preparation stage.

[0180] 3. Text-to-sound conversion task:

[0181] Input: Instructions describing the scenario and events. Content: By target duration. calculate And construct the [SFX] repeating sequence.

[0182] System execution: No G2P required. Special symbols are mapped via an embedding table before entering the content encoder.

[0183] The above embodiments can use natural language commands as the high-level control entry point and content sequences to carry the temporal structure, reducing system integration and product integration costs. Furthermore, it reduces reliance on reference audio for modalities such as speech, which is beneficial for batch generation and cold start scenarios. Dynamic injection of special symbols allows sound effects to enter a sequence conditional channel isomorphic to phonemes, enabling generation under a unified attention mechanism. This avoids architectural fragmentation caused by adding separate mutually exclusive branches for sound effects. A data-driven scaling relationship is established between the length of the sound effect content sequence and the target duration, allowing the model to obtain explicit temporal anchors and improving the consistency of long sequence unfolding. Phased course learning alleviates optimization conflicts in multimodal joint training, reduces the risk of negative transfer, and minimizes damage to speech intelligibility and musical structure. Under unified training conditions, the speech side achieves more robust acoustic representation learning, while the music side achieves better long-term consistency-related benefits.

[0184] It should be understood that although the steps in the flowchart of this disclosure are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless expressly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the diagram may include multiple steps or multiple stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0185] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0186] Figure 6 This is a block diagram illustrating an audio generation model processing apparatus according to an exemplary embodiment. (Refer to...) Figure 6 The device includes a training sample acquisition unit 601, a sample encoding acquisition unit 602, a sample encoding processing unit 603, a vector field construction unit 604, and a generative model training unit 605.

[0187] The training sample acquisition unit 601 is configured to acquire the sample task instruction text, sample task content text, and target audio that matches the sample audio generation task corresponding to the sample audio generation task; the sample audio generation task includes at least two task types.

[0188] The sample encoding acquisition unit 602 is configured to input the sample task instruction text, the sample task content text, and the target audio into the audio generation model to be trained, and generate the instruction encoding corresponding to the sample task instruction text, the content encoding corresponding to the sample task content text, and the target audio encoding corresponding to the target audio.

[0189] The sample encoding processing unit 603 is configured to concatenate the instruction encoding and the content encoding to generate the task text encoding corresponding to the sample audio generation task, and to add noise to the target audio encoding using noise samples according to the time parameters associated with the sample audio generation task to obtain the noise-added audio encoding.

[0190] The vector field construction unit 604 is configured to perform task text encoding, time parameters and noise-added frequency encoding to generate a predicted vector field, and to construct a real vector field based on the difference between the noise sample and the target audio encoding.

[0191] Generative model training unit 605 is configured to train an audio generation model based on the difference between the predicted vector field and the true vector field to obtain a trained audio generation model.

[0192] In an exemplary embodiment, the content encoding is generated based on the phoneme sequence of the sample task content text. When the sample audio generation task is a text-to-sound effect task, the training sample acquisition unit 601 is further configured to acquire a preset global scaling factor. The global scaling factor is used to characterize the ratio between the phoneme sequence length and the audio duration. Based on the audio duration of the target audio and the global scaling factor, the phoneme sequence length corresponding to the sample task content text is acquired. Repeated preset symbol units are generated according to the phoneme sequence length corresponding to the sample task content text, and the repeated preset symbol units are used as the phoneme sequence of the sample task content text.

[0193] In an exemplary embodiment, the training sample acquisition unit 601 is further configured to acquire the phoneme sequence length corresponding to each preset speech sample and the audio duration corresponding to each preset speech sample; acquire the ratio between the phoneme sequence length and the audio duration corresponding to each preset speech sample based on the phoneme sequence length and the audio duration; and use the average value of each ratio as a global scaling factor.

[0194] In one exemplary embodiment, the training of the audio generation model includes multiple training phases; the training sample acquisition unit 601 is further configured to acquire sample audio generation tasks matching the current training phase, as well as sample task instruction text, sample task content text, and target audio matching the sample audio generation task; the sample encoding acquisition unit 602 is further configured to input the sample task instruction text, sample task content text, and target audio into the audio generation model matching the current training phase; the generation model training unit 605 is further configured to, if the current training phase is not the last of multiple training phases, take the next training phase of the current training phase as the new current training phase, and take the audio generation model as the new audio generation model matching the current training phase, and return to the step of acquiring sample audio generation tasks matching the current training phase, until the current training phase is the last of multiple training phases, and take the audio generation model as the trained audio generation model.

[0195] In one exemplary embodiment, the sample audio generation task includes a text-to-speech task, a text-to-music task, and a text-to-sound effects task. Multiple training phases include a first training phase, a second training phase, and a third training phase executed sequentially. The training sample acquisition unit 601 is further configured to, when the current training phase is the first training phase, assign the sample audio generation task matched in the current training phase as a first sample audio generation task; the first sample audio generation task includes a text-to-speech task; when the current training phase is the second training phase, assign the sample audio generation task matched in the current training phase as a second sample audio generation task; the second sample audio generation task includes a text-to-speech task and a text-to-music task; when the current training phase is the third training phase, assign the sample audio generation task matched in the current training phase as a third sample audio generation task; the third sample audio generation task includes a text-to-speech task, a text-to-music task, and a text-to-sound effects task.

[0196] In one exemplary embodiment, the vector field construction unit 604 is further configured to perform a joint diffusion layer of the audio generation model that inputs the task text code, time parameters, and noise-added frequency code into the audio generation model; perform joint attention processing on the task text code and noise-added frequency code using the time parameters to obtain an audio code aligned with the task text code; input the audio code aligned with the time parameters and the task text code into a single-path diffusion layer of the audio generation model; perform self-attention refinement processing on the audio code aligned with the task text code using the time parameters to obtain a refined audio code; and input the refined audio code into the output layer of the audio generation model to obtain a predicted vector field.

[0197] Figure 7This is a block diagram illustrating an audio generation apparatus according to an exemplary embodiment. (Refer to...) Figure 7 The device includes a task information acquisition unit 701, a task encoding acquisition unit 702, a vector field prediction unit 703, and a target audio generation unit 704.

[0198] The task information acquisition unit 701 is configured to acquire the task instruction text corresponding to the audio generation task to be processed, and the task content text corresponding to the audio generation task to be processed.

[0199] The task encoding acquisition unit 702 is configured to execute the audio generation model after inputting the task instruction text, the task content text, and the preset noise signal into the trained audio generation model, and generate the instruction encoding corresponding to the task instruction text, the content encoding corresponding to the task content text, and the noise audio encoding corresponding to the preset noise signal; the audio generation model is trained by the audio generation model training method of any of the above embodiments.

[0200] The vector field prediction unit 703 is configured to concatenate the instruction encoding and the content encoding to generate the task text encoding corresponding to the audio generation task to be processed, and generate the prediction vector field corresponding to the audio generation task to be processed based on the task text encoding, the pre-set time parameters and the noise audio encoding.

[0201] The target audio generation unit 704 is configured to perform numerical integration on the prediction vector field to obtain the target audio encoding, and to decode the target audio encoding to obtain the target audio corresponding to the audio generation task to be processed.

[0202] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0203] Figure 8 This is a block diagram illustrating an electronic device 800 for audio generation according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0204] Reference Figure 8 The electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816.

[0205] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0206] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of such data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, optical disk, or graphene memory.

[0207] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0208] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0209] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0210] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0211] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 can detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or its components, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0212] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0213] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0214] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above-described method. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0215] In an exemplary embodiment, a computer program product is also provided, the computer program product including instructions that can be executed by a processor 820 of an electronic device 800 to perform the above-described method.

[0216] It should be noted that the above-mentioned apparatus, electronic equipment, computer-readable storage medium, computer program product, etc., may also include other implementation methods according to the description of the method embodiments. For specific implementation methods, please refer to the description of the relevant method embodiments, which will not be elaborated here.

[0217] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0218] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An audio generation model processing method, characterized in that, include: Obtain the sample task instruction text, sample task content text, and target audio that matches the sample audio generation task corresponding to the sample audio generation task; the sample audio generation task includes at least two task types; The sample task instruction text, the sample task content text, and the target audio are input into the audio generation model to be trained to generate the instruction code corresponding to the sample task instruction text, the content code corresponding to the sample task content text, and the target audio code corresponding to the target audio. The instruction encoding and the content encoding are concatenated to generate the task text encoding corresponding to the sample audio generation task. Then, according to the time parameters associated with the sample audio generation task, the target audio encoding is noise-added using noise samples to obtain noise-added audio encoding. Based on the task text encoding, the time parameters, and the noise-added audio encoding, a prediction vector field is generated, and based on the difference between the noise samples and the target audio encoding, a true vector field is constructed. The audio generation model is trained based on the difference between the predicted vector field and the true vector field to obtain the trained audio generation model.

2. The method according to claim 1, characterized in that, The content encoding is generated based on the phoneme sequence of the sample task content text. In the case where the sample audio generation task is a text-to-sound effect task, the phoneme sequence of the sample task content text is obtained through the following steps: Get the preset global scaling factor; The global scaling factor is used to characterize the ratio between the phoneme sequence length and the audio duration. Based on the audio duration of the target audio and the global scaling factor, obtain the phoneme sequence length corresponding to the sample task content text; A set of repeating preset symbol units is generated according to the length of the phoneme sequence corresponding to the sample task content text, and the repeating preset symbol units are used as the phoneme sequence of the sample task content text.

3. The method according to claim 2, characterized in that, The global scaling factor is obtained through the following steps: Obtain the phoneme sequence length corresponding to each preset speech sample, and the audio duration corresponding to each preset speech sample; Based on the length of each phoneme sequence and the duration of the audio, obtain the ratio between the length of the phoneme sequence and the duration of the audio corresponding to each preset speech sample; The average of the ratios is used as the global scaling factor.

4. The method according to claim 1, characterized in that, The training of the audio generation model includes multiple training stages; the acquisition of the sample task instruction text, sample task content text, and target audio matching the sample audio generation task corresponding to the sample audio generation task includes: Obtain the sample audio generation task that matches the current training phase, as well as the sample task instruction text, sample task content text, and target audio that match the sample audio generation task. The step of inputting the sample task instruction text, the sample task content text, and the target audio into the audio generation model to be trained includes: The audio generation model that matches the sample task instruction text, the sample task content text, and the target audio input with the current training phase. After training the audio generation model, the method further includes: If the current training phase is not the last of the plurality of training phases, the next training phase of the current training phase is taken as the new current training phase, and the audio generation model is taken as the audio generation model matched by the new current training phase. The step of obtaining sample audio generation task matched with the current training phase is returned until the current training phase is the last of the plurality of training phases, and the audio generation model is taken as the trained audio generation model.

5. The method according to claim 4, characterized in that, The sample audio generation task includes text-to-speech, text-to-music, and text-to-sound effects tasks. The multiple training stages include a first training stage, a second training stage, and a third training stage executed sequentially. The step of obtaining the sample audio generation task matching the current training stage includes: When the current training phase is the first training phase, the sample audio generation task matched by the current training phase is the first sample audio generation task; the first sample audio generation task includes the text-to-speech task. When the current training phase is the second training phase, the sample audio generation task matched by the current training phase is the second sample audio generation task; the second sample audio generation task includes the text-to-speech task and the text-to-music task. When the current training phase is the third training phase, the sample audio generation task matched in the current training phase is the third sample audio generation task; the third sample audio generation task includes the text-to-speech task, the text-to-music task, and the text-to-sound effects task.

6. The method according to any one of claims 1 to 5, characterized in that, The step of generating a prediction vector field based on the task text encoding, the time parameters, and the noise-added frequency encoding includes: The task text encoding, the time parameter, and the noise-added frequency encoding are input into the joint diffusion layer of the audio generation model. The time parameter is used to perform joint attention processing on the task text encoding and the noise-added frequency encoding to obtain an audio encoding aligned with the task text encoding. The time parameters and the audio code aligned with the task text code are input into the single-path diffusion layer of the audio generation model. The time parameters are used to perform self-attention refinement processing on the audio code aligned with the task text code to obtain a refined audio code. The refined audio encoding is input into the output layer of the audio generation model to obtain the prediction vector field.

7. An audio generation method, characterized in that, include: Obtain the task instruction text corresponding to the audio generation task to be processed, and the task content text corresponding to the audio generation task to be processed; The task instruction text, task content text, and preset noise signal are input into the trained audio generation model to generate the instruction code corresponding to the task instruction text, the content code corresponding to the task content text, and the noise audio code corresponding to the preset noise signal. The audio generation model is trained using the audio generation model training method as described in any one of claims 1 to 6; The instruction encoding and the content encoding are concatenated to generate the task text encoding corresponding to the audio generation task to be processed. Based on the task text encoding, the preset time parameters and the noise audio encoding, the prediction vector field corresponding to the audio generation task to be processed is generated. The predicted vector field is numerically integrated to obtain the target audio code, and the target audio code is decoded to obtain the target audio corresponding to the audio generation task to be processed.

8. An audio generation model processing device, characterized in that, include: The training sample acquisition unit is configured to acquire the sample task instruction text, sample task content text, and target audio matching the sample audio generation task corresponding to the sample audio generation task; the sample audio generation task includes at least two task types. The sample encoding acquisition unit is configured to take the sample task instruction text, the sample task content text, and the target audio as inputs to the audio generation model to be trained, and generate the instruction encoding corresponding to the sample task instruction text, the content encoding corresponding to the sample task content text, and the target audio encoding corresponding to the target audio. The sample encoding processing unit is configured to concatenate the instruction encoding with the content encoding to generate the task text encoding corresponding to the sample audio generation task, and to add noise to the target audio encoding using noise samples according to the time parameters associated with the sample audio generation task to obtain noise-added audio encoding. The vector field construction unit is configured to generate a predicted vector field based on the task text encoding, the time parameters, and the noise-added audio encoding, and to construct a true vector field based on the difference between the noise samples and the target audio encoding. The generative model training unit is configured to train the audio generation model based on the difference between the predicted vector field and the true vector field to obtain the trained audio generation model.

9. An audio generation device, characterized in that, include: The task information acquisition unit is configured to acquire the task instruction text corresponding to the audio generation task to be processed, and the task content text corresponding to the audio generation task to be processed. The task encoding acquisition unit is configured to execute the trained audio generation model by inputting the task instruction text, the task content text, and the preset noise signal, and generate the instruction encoding corresponding to the task instruction text, the content encoding corresponding to the task content text, and the noise audio encoding corresponding to the preset noise signal. The audio generation model is trained using the audio generation model training method as described in any one of claims 1 to 6; The vector field prediction unit is configured to concatenate the instruction encoding and the content encoding to generate the task text encoding corresponding to the audio generation task to be processed, and generate the prediction vector field corresponding to the audio generation task to be processed based on the task text encoding, the preset time parameters and the noise audio encoding. The target audio generation unit is configured to perform numerical integration on the prediction vector field to obtain the target audio encoding, and to decode the target audio encoding to obtain the target audio corresponding to the audio generation task to be processed.

10. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the audio generation model processing method as described in any one of claims 1 to 6, or the audio generation method as described in claim 7.

11. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the audio generation model processing method as described in any one of claims 1 to 6, or the audio generation method as described in claim 7.

12. A computer program product, the computer program product comprising instructions, characterized in that, When the instruction is executed by the processor of the electronic device, the electronic device is able to perform the audio generation model processing method as described in any one of claims 1 to 6, or the audio generation method as described in claim 7.

Images