An audio generation method and apparatus based on an audio processing model

Abstract

The application discloses an audio generation method and device based on an audio processing model, and the method comprises the following steps: determining a historical audio sequence or a historical audio sequence and a current audio frame text vector, performing autoregressive processing on the historical audio sequence to obtain a context vector sequence; wherein the historical audio sequence is composed of at least two historical audio frames; and the audio processing model performs denoising processing on a random noisy audio vector to be processed based on the context vector sequence or the context vector sequence and the current audio frame text vector to obtain a current audio frame; the current audio frame is added to the end of the historical audio sequence, and the steps of determining the historical audio sequence or the historical audio sequence and the current audio frame text vector are repeatedly executed until the processing of all audio frames is completed. The application can process a streaming input based on the audio processing model, realize streaming audio generation, improve the real-time performance of audio generation, and ensure good audio listening experience.

Description

Technical Field

[0001] This invention relates to the field of audio processing technology, and in particular to an audio generation method and apparatus based on an audio processing model. Background Technology

[0002] In audio generation scenarios, non-streaming generation schemes are typically used, where the audio processing model receives all the audio features to be synthesized before starting to synthesize the corresponding audio. However, in scenarios with high real-time requirements, non-streaming generation schemes cannot meet the immediacy requirements. Accordingly, to meet the requirements of real-time scenarios, existing technologies first divide the input audio features into blocks, and then process them sequentially according to the block order, simulating the effect of audio streaming output. With this block-based approach, the more blocks there are, the more times the audio processing model is called, and the greater the model's computational overhead, but it can achieve higher real-time output performance. However, if the length of each block is longer, the audio generation effect is better. Therefore, a longer block length is usually set to reduce the number of blocks. This leads to higher audio generation latency, and the transitions between blocks may be abrupt, affecting the listening experience. Therefore, there is an urgent need for a streaming audio generation scheme that can meet the requirements of real-time scenarios. Summary of the Invention

[0003] This invention provides an audio generation method and apparatus based on an audio processing model, which enables streaming audio generation by processing streaming input based on the audio processing model, thereby improving the real-time performance of audio generation while ensuring good audio quality.

[0004] In a first aspect, embodiments of the present invention provide an audio generation method based on an audio processing model, the method comprising:

[0005] Determine a historical audio sequence or a historical audio sequence and the text vector of the current audio frame, and perform autoregressive processing on the historical audio sequence to obtain a context vector sequence;

[0006] The historical audio sequence consists of at least two historical audio frames;

[0007] The audio processing model performs noise reduction processing on the random noisy frequency vector to be processed based on the context vector sequence or the context vector sequence and the current audio frame text vector to obtain the current audio frame.

[0008] The current audio frame is added to the end of the historical audio sequence, and the steps of determining the historical audio sequence or the text vector of the historical audio sequence and the current audio frame are repeated until the processing of all audio frames is completed.

[0009] Optionally, the historical audio sequence is subjected to autoregressive processing to obtain a context vector sequence, including:

[0010] The target audio frame is converted into a target floating-point vector using an autoregressive sequence processing model.

[0011] A positional encoding is added to the target floating-point vector, the positional encoding being used to represent the position of the target audio frame in the historical audio sequence;

[0012] Based on the causal attention mechanism, the association information between the target floating-point vector and the historical floating-point vectors corresponding to the historical audio frames preceding the target audio frame is determined;

[0013] Information fusion is performed based on the associated information to obtain a global context vector that matches the target audio frame;

[0014] The context vector sequence is obtained based on the global context vector of each audio frame.

[0015] Optionally, the audio processing model performs denoising processing on the random noisy frequency vector to be processed based on the context vector sequence or the context vector sequence and the current audio frame text vector to obtain the current audio frame, including:

[0016] Using a diffusion model and masking mechanism, the random noisy frequency vector to be processed is denoised based on the context vector sequence or the context vector sequence and the current audio frame text vector to obtain the current audio frame.

[0017] In the diffusion model, when denoising the random noisy frequency vector to be processed, the denoising range is limited to the current audio frame through a masking mechanism.

[0018] Optionally, the training process of the diffusion model includes:

[0019] Obtain the sample audio vector sequence;

[0020] Determine the position to be predicted in the sample audio vector sequence, add a mask to the position to be predicted and the sample audio vectors after the position to be predicted, and obtain the masked sample audio vector sequence.

[0021] A pre-defined time-step noise is superimposed on each sample audio vector in the masked sample audio vector sequence to obtain a noisy masked sample audio vector sequence.

[0022] At least the masked sample audio vector sequence is input into a pre-set diffusion model, and the pre-set diffusion model performs denoising processing on the masked sample audio vector sequence in parallel;

[0023] Based on the predicted denoising results corresponding to each masked sample audio vector sequence and the masked sample audio vector sequence before adding noise, the model loss is determined, and the model parameters are adjusted based on backpropagation of the model loss.

[0024] Repeatedly and randomly execute the operation to determine the position to be predicted in the sample audio vector sequence until the preset training conditions are met, and obtain the trained diffusion model.

[0025] Optionally, the audio processing model, based on the context vector sequence or the context vector sequence and the current audio frame text vector, performs denoising processing on the random noisy frequency vector to be processed to obtain the current audio frame, and further includes:

[0026] Using an autoregressive diffusion model, based on the context vector sequence or the context vector sequence and the current audio frame text vector, the random noisy frequency vector to be processed is subjected to autoregressive denoising processing to obtain the current audio frame.

[0027] Optionally, the training steps of the autoregressive diffusion model include:

[0028] By superimposing noise at a preset time step onto the sample audio vectors, a noisy sample audio vector sequence is obtained;

[0029] The position to be predicted in the sample audio vector sequence is determined, and the noisy sample audio vector sequence is input into a preset diffusion model. The preset diffusion model is then used to predict the prediction noise corresponding to the position to be predicted based on a causal attention mechanism.

[0030] Based on the predicted noise and the actual noise superimposed on the sample audio vectors, the model loss is calculated, and the model parameters are adjusted through backpropagation.

[0031] Repeat the above training steps until the autoregressive diffusion model meets the preset training conditions, and the trained autoregressive diffusion model is obtained.

[0032] Optionally, before the audio processing model performs audio processing, a historical audio sequence or a historical audio sequence and the current audio frame text vector are determined, including:

[0033] The text to be processed is converted into speech using a speech conversion model to obtain a historical audio sequence, or a historical audio sequence and a text vector of the current audio frame. The text to be processed includes a textual description of the audio frame and / or the speech content.

[0034] Alternatively, the text to be processed can be processed using a music generation model to obtain a historical audio sequence, or a historical audio sequence and a text vector of the current audio frame, wherein the text to be processed includes a textual description of the audio frame and / or lyrics.

[0035] Optionally, the text to be processed is converted into speech using a speech conversion model to obtain a historical audio sequence, or a historical audio sequence and a text vector of the current audio frame, including:

[0036] If the text to be processed includes textual descriptions, then the textual descriptions are converted into speech conditional vectors, and at least one text vector sequence is generated based on the text to be processed using a text encoder, or an artificial intelligence model and a text encoder.

[0037] Obtain an autoregressive acoustic model, input the text vector sequence into the autoregressive acoustic model, and let the autoregressive acoustic model generate historical audio sequences, or historical audio sequences and current audio frame text vectors, step by step based on the text vector sequence.

[0038] Alternatively, the text vector sequence and the speech condition vector can be input into the autoregressive acoustic model, causing the autoregressive acoustic model to autoregressively generate historical audio sequences, or historical audio sequences and current audio frame text vectors, based on the text vector sequence and the condition vector.

[0039] Optionally, the text to be processed is processed using a music generation model to obtain a historical audio sequence, or a historical audio sequence and a text vector of the current audio frame, including:

[0040] If the text to be processed includes textual descriptions, then based on an artificial intelligence model, the textual descriptions are converted into music conditional vectors, and based on a text encoder, or an artificial intelligence model and a text encoder, at least one music vector sequence is generated based on the text to be processed.

[0041] Obtain an autoregressive music generation model, input the music vector sequence into the autoregressive music generation model, and let the autoregressive music generation model generate historical audio sequences or historical audio sequences and current audio frame text vectors step by step based on the music vector sequence.

[0042] Alternatively, the music vector sequence and the music condition vector can be input into the autoregressive music generation model, so that the autoregressive music generation model can generate historical audio sequences, or historical audio sequences and current audio frame text vectors, step by step based on the music vector sequence and the music condition vector.

[0043] The historical audio sequence includes at least two channels, and audio frames at the same position in at least two channels are generated in parallel.

[0044] Secondly, embodiments of the present invention also provide an audio generation apparatus based on an audio processing model, the apparatus comprising:

[0045] The context vector sequence determination module is used to determine a historical audio sequence or a historical audio sequence and the current audio frame text vector, and to perform autoregressive processing on the historical audio sequence to obtain a context vector sequence.

[0046] The historical audio sequence consists of at least two historical audio frames;

[0047] The current audio frame determination module is used by the audio processing model to perform noise reduction processing on the random noisy frequency vector to be processed based on the context vector sequence or the context vector sequence and the current audio frame text vector, so as to obtain the current audio frame;

[0048] The audio sequence update module is used to add the current audio frame to the end of the historical audio sequence and repeatedly execute the step of determining the historical audio sequence or the text vector of the historical audio sequence and the current audio frame until the processing of all audio frames is completed.

[0049] The technical solution of this invention involves performing autoregressive processing on a historical audio sequence composed of currently generated historical audio frames to obtain a context vector sequence. This context vector sequence, or a combination of the context vector sequence and the text vector of the current audio frame, is used as input to an audio processing model. The model uses the input data as a condition vector to denoise the random, noisy frequency vectors to be processed, resulting in the current audio frame. This current audio frame is then inserted at the end of the historical audio sequence and used again as the historical audio sequence for generating the next audio frame. This process is repeated until all audio frames are generated. This technical solution directly performs processing operations on the audio frames, avoiding audio encoding and decoding operations and reducing audio processing overhead. It also enables processing based on streaming audio data, achieving streaming audio generation frame by frame, improving the real-time performance of audio generation. Simultaneously, it ensures the semantic coherence of the audio, enhancing the listening experience.

[0050] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart of an audio generation method based on an audio processing model provided in Embodiment 1 of the present invention;

[0053] Figure 2This is a flowchart of another audio generation method based on an audio processing model provided in Embodiment 2 of the present invention;

[0054] Figure 3 This is a flowchart of another audio generation method based on an audio processing model provided in Embodiment 3 of the present invention;

[0055] Figure 4 This is a schematic diagram of the structure of an audio generation device based on an audio processing model provided in Embodiment 4 of the present invention;

[0056] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. Detailed Implementation

[0057] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0058] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices. In the embodiments of this application, certain software, components, models, and other existing industry solutions may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solutions of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0059] The acquisition, transmission, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0060] Example 1:

[0061] Figure 1The flowchart of an audio generation method based on an audio processing model is provided in Embodiment 1 of the present invention. This embodiment is applicable to the generation of streaming audio based on an audio processing model. The method can be executed by an audio generation device based on an audio processing model. The audio generation device based on an audio processing model can be implemented in hardware and / or software and can be configured in a server or electronic device.

[0062] like Figure 1 As shown, the method includes:

[0063] S110. Determine the historical audio sequence or the historical audio sequence and the current audio frame text vector, and perform autoregressive processing on the historical audio sequence to obtain the context vector sequence.

[0064] The historical audio sequence consists of at least two historical audio frames, and the current audio frame text vector indicates the text features contained in the current audio frame. Historical audio frames are audio frames that have been generated sequentially and formed into a sequence before the current time. An audio frame is the basic unit used in audio coding to represent a small segment of audio data; each audio frame contains a certain number of audio samples.

[0065] In this embodiment, the audio sequence generated by streaming can be processed sequentially, thereby improving the real-time performance and coherence of the processed audio and enhancing the listening experience.

[0066] Context vector sequences are achieved by capturing the temporal dependencies between audio frames in a historical audio sequence (such as intonation changes in speech and melodic transitions in music) and global semantics, enabling information coordination and aggregation among multiple historical audio frames. In this embodiment, the process of obtaining a context vector sequence based on the historical audio sequence is a frame-by-frame processing and dynamic transmission of audio information. Each output new context vector sequence contains the features of the latest audio frame and aggregated information of the features of all historical audio frames.

[0067] Obtaining a context vector sequence based on a historical audio sequence can be achieved using models such as recurrent neural networks, bidirectional recurrent networks, transformer encoders based on self-attention mechanisms, or cross-attention mechanisms. It can also be achieved using deep convolutional networks, dilated convolutional networks, etc. This invention does not limit the specific methods used. In this embodiment, the process of obtaining a context vector sequence based on a historical audio sequence through autoregressive processing is used as an example to illustrate the specific steps involved.

[0068] Specifically, autoregressive processing of historical audio sequences yields a context vector sequence, which can include:

[0069] S10. Convert the target audio frame into a target floating-point vector using an autoregressive sequence processing model.

[0070] The autoregressive sequence processing model can be an AR (Autoregressive) Transformer model. The input to the autoregressive sequence processing model is discrete historical audio frames, and the output is a continuous sequence of context vectors. The target audio frame refers to the historical audio frame that the autoregressive sequence processing model is currently processing. In this embodiment, the embedding layer of the autoregressive sequence processing model converts the target audio frame into a target floating-point vector.

[0071] S20. Add position encoding to the target floating-point vector, the position encoding being used to represent the position of the target audio frame in the historical audio sequence.

[0072] The positional encoding can be determined either based on the order in which the autoregressive sequence processing model receives the target audio frames, in which case the positional encoding represents the relative position of the target audio frame in the historical audio sequence; or based on the natural time of receipt of the target audio frames by the autoregressive sequence processing model, in which case the positional encoding represents the absolute position of the target audio frame in the historical audio sequence.

[0073] S30. Based on the causal attention mechanism, determine the association information between the target floating-point vector and the historical floating-point vectors corresponding to the historical audio frames preceding the target audio frame.

[0074] For autoregressive sequence processing models, based on causal attention mechanisms, they can capture the correlation information between the target floating-point vector corresponding to the target audio frame being processed and the historical floating-point vectors corresponding to the historical audio frames preceding the target audio frame, including semantic relationships and dependencies.

[0075] S40. Based on the associated information, information fusion is performed to obtain a global context vector that matches the target audio frame.

[0076] The global context vector matched with the target audio frame contains association information of all historical audio frames from the first frame to the target audio frame, and incorporates temporal dependencies at different scales.

[0077] Specifically, based on the associated information corresponding to the target audio frame, the attention score between the target audio frame and its previous historical audio frames is calculated, and the attention weight is further calculated. Finally, the attention scores are weighted and summed according to the attention weight to obtain the global context vector of the target audio frame.

[0078] S50. Obtain the context vector sequence based on the global context vector of each audio frame.

[0079] The global context vectors of each audio frame form a context vector sequence, which is the same length as the historical audio sequence input to the autoregressive sequence processing model. Within this context vector sequence, each global context vector is a condensed encoding of the global preceding context information up to that historical audio frame.

[0080] In this embodiment, the autoregressive sequence processing model is a streaming sequence model that can process received streaming audio frames in a causal pattern.

[0081] Furthermore, autoregressive sequence processing models, such as AR Transformer models, can be trained using conventional AR Transformer model training methods. For example, audio data can be input into the model to be trained, and a lower triangular mask can be used to limit the range of self-attention. Then, through backpropagation, the parameters of each module can be fine-tuned based on loss functions such as cross-entropy. This embodiment will not elaborate on this.

[0082] Furthermore, before the audio processing model performs audio processing, determining the historical audio sequence or the historical audio sequence and the current audio frame text vector may include:

[0083] S1. Using a speech conversion model, the text to be processed is converted into speech to obtain a historical audio sequence, or a historical audio sequence and a text vector of the current audio frame. The text to be processed includes a textual description of the audio frame and / or the speech content.

[0084] S2. Alternatively, the text to be processed is processed through a music generation model to obtain a historical audio sequence, or a historical audio sequence and a text vector of the current audio frame, wherein the text to be processed includes a textual description of the audio frame and / or lyrics.

[0085] This embodiment can be applied to text-driven speech generation scenarios, as well as music generation scenarios, etc. In text-driven speech generation scenarios, a historical audio sequence, or a historical audio sequence and the current audio frame text vector, is generated based on a speech conversion model; in music generation scenarios, a historical audio sequence, or a historical audio sequence and the current audio frame text vector, is generated based on a music generation model.

[0086] The speech conversion model can be a TTS (Text-to-Speech) model. For text-driven speech generation scenarios, the text to be processed can be a textual description, such as timbre, accent, emotion, speech rate, etc.; it can also be speech content, that is, the text content corresponding to the speech to be generated; or it can be a combination of textual description and speech content. In this embodiment, the content form of the text to be processed is not limited.

[0087] The music generation model can be based on recurrent neural networks, transformers, generative adversarial networks, or diffusion models, etc. This embodiment does not limit the type of music generation model. For the music generation scenario, the music can be in various forms, such as instrumental music, vocal music, or a combination of instrumental music and vocals. If it is a combination of instrumental music and vocals, the instrumental music and vocals correspond to different audio tracks, and these tracks are independent of each other. Correspondingly, for music, the text to be processed can be a textual description of the audio frames, such as tempo, pitch, rhythm, chords, instruments, or genre; for vocals, the text to be processed can be a textual description of the audio frames, such as the singer's timbre and pitch, or it can include lyrics.

[0088] In this embodiment, since the input to the subsequent audio processing model can be either a sequence of context vectors or a sequence of context vectors and the current audio frame text vector, it can be flexibly set according to the actual needs of the business scenario. Therefore, in this step, depending on the input to the subsequent audio processing model, it can be determined whether the speech conversion model or the music generation model outputs a sequence of context vectors or a sequence of context vectors and the current audio frame text vector.

[0089] Furthermore, this embodiment can also be applied to scenarios such as audio completion, expansion, or repair. In the above scenarios, the existing audio to be completed, expanded, or repaired can be directly used as a historical audio sequence.

[0090] Furthermore, before the audio processing model performs audio processing, S1 may include:

[0091] S11. If the text to be processed includes textual descriptions, the textual descriptions are converted into speech conditional vectors, and at least one text vector sequence is generated based on the text to be processed using a text encoder, or an artificial intelligence model and a text encoder.

[0092] S12. Obtain the autoregressive acoustic model, input the text vector sequence into the autoregressive acoustic model, and let the autoregressive acoustic model generate historical audio sequences or historical audio sequences and current audio frame text vectors step by step based on the text vector sequence.

[0093] S13, or, input the text vector sequence and the speech condition vector into the autoregressive acoustic model, so that the autoregressive acoustic model generates a historical audio sequence, or a historical audio sequence and the current audio frame text vector, in an autoregressive manner based on the text vector sequence and the condition vector.

[0094] In text-driven speech generation scenarios, the text to be processed can include textual descriptions and / or speech content. For textual descriptions, they are converted into speech conditional vectors, which serve as input to the subsequent autoregressive acoustic model. By introducing an artificial intelligence model, the speech content matching the textual description is predicted, and the speech content is converted into a text vector sequence using a text encoder. For example, the textual description could be "Xiaoming happily told his mother a little story that happened at school today." The artificial intelligence model can expand Xiaoming's speech content into a short story based on the above description and interpret it based on parameters such as timbre associated with Xiaoming in the context. For speech content, it can be directly converted into a text vector sequence using a text encoder. When the text to be processed includes both textual descriptions and speech content, the above processing methods can be applied to the textual descriptions and speech content respectively.

[0095] The autoregressive acoustic model is an autoregressive model that supports text-to-speech (TTS). When the text to be processed is speech content, the text vector sequence obtained by the text encoder converting the speech content is used as the input of the autoregressive acoustic model.

[0096] When the text to be processed is a textual description, the text vector sequence after the artificial intelligence model converts the speech content predicted by the textual description, and the speech condition vector obtained by converting the textual description, are used as the input of the autoregressive acoustic model.

[0097] When the text to be processed includes both textual descriptions and speech content, the text vector sequence obtained by the artificial intelligence model based on the speech content predicted by the textual description and then transformed, the speech condition vector obtained by the transformation of the textual description, and the text vector sequence obtained by the text encoder directly transforming the speech content are used as inputs to the autoregressive acoustic model.

[0098] Furthermore, S2 can include:

[0099] S21. If the text to be processed includes textual descriptions, then based on an artificial intelligence model, the textual descriptions are converted into music conditional vectors, and based on a text encoder, or an artificial intelligence model and a text encoder, at least one music vector sequence is generated based on the text to be processed.

[0100] S22. Obtain the autoregressive music generation model, input the music vector sequence into the autoregressive music generation model, and let the autoregressive music generation model generate historical audio sequences or historical audio sequences and current audio frame text vectors step by step based on the music vector sequence.

[0101] S23, or, input the music vector sequence and the music condition vector into the autoregressive music generation model, so that the autoregressive music generation model generates a historical audio sequence, or a historical audio sequence and the current audio frame text vector, step by step based on the music vector sequence and the music condition vector;

[0102] In music generation scenarios, the text to be processed may include textual descriptions and / or lyrics. Similarly, for textual descriptions, they are converted into music conditional vectors, which serve as input to the subsequent autoregressive music generation model. Simultaneously, an artificial intelligence model is introduced to predict the matching lyrics based on the textual descriptions, and a text encoder converts the lyrics into a music vector sequence. For example, the textual description could be "Xiaoming happily sang a children's song," and the AI ​​model can predict Xiaoming's lyrics. The lyrics can be directly converted into a music vector sequence using the text encoder. When the text to be processed includes both textual descriptions and lyrics, the above processing methods are applied separately to the textual descriptions and lyrics.

[0103] When the text to be processed is lyrics, the music vector sequence obtained by the text encoder is used as the input to the autoregressive music generation model. When the text to be processed is a textual description, the music vector sequence obtained by the AI ​​model after further transformation of the lyrics based on the textual description, as well as the music condition vector obtained by the textual description transformation, are used as the input to the autoregressive acoustic model. When the text to be processed contains both textual description and lyrics, the music vector sequence obtained by the AI ​​model after further transformation of the lyrics based on the textual description, the music condition vector obtained by the textual description transformation, and the music vector sequence obtained by the text encoder directly transforming the lyrics are used as the input to the autoregressive acoustic model.

[0104] Furthermore, when the music includes both vocals and accompaniment, the historical audio sequence includes at least two channels, and audio frames at the same position in at least two channels are generated in parallel. Different channels represent multiple different audio tracks, such as vocal parts and musical accompaniment parts. Further, it can also include drum tracks, instrument tracks, etc., or for the musical accompaniment part, different instruments can each correspond to a separate track, or for the vocal part, in a chorus, each person corresponds to a separate track, ensuring that different parts of the music do not mix. The number of channels corresponding to the historical audio sequence, and the content corresponding to each channel, can be determined in the textual description.

[0105] Understandably, since this embodiment is a streaming process, that is, generating and updating historical audio frames in real time and updating the historical audio sequence, and updating the context vector sequence in real time, the context vector sequence will better capture long-term dependencies in the audio as the number of historical audio frames in the historical audio sequence increases. Therefore, whether it is a text-driven speech generation scenario, a music generation scenario, or an audio completion, expansion, or repair scenario, the historical audio sequence generated based on the speech conversion model, the historical audio sequence generated based on the music generation model, or the existing audio to be completed, expanded, or repaired can be used as the initial historical audio sequence in the above embodiment. When the number of historical audio frames in the initial historical audio sequence is greater than or equal to a preset value, the context vector sequence is generated according to the streaming process of this embodiment, and new audio frames are obtained by denoising through the audio processing model and added to the initial historical audio sequence, updating the historical audio sequence. The streaming process of this embodiment is repeated to continuously update the historical audio sequence until all audio frames are processed.

[0106] S120. The audio processing model performs noise reduction processing on the random noisy frequency vector to be processed based on the context vector sequence or the context vector sequence and the current audio frame text vector to obtain the current audio frame.

[0107] The audio processing model is based on the diffusion model. The random noisy frequency vector is generated by the audio processing model through a forward diffusion process. During denoising, the audio processing model starts from the random noisy frequency vector and performs reverse diffusion at each time step from T to 1, that is, predicting and removing noise, ultimately obtaining a clear and audible audio frame, i.e., the current audio frame. Here, T represents the maximum number of iterations in the denoising step of the diffusion model, which is preset by the user. In the forward noise addition process, it controls the total number of steps in which the original data is gradually denoised; by step T, the data is close to isotropic Gaussian noise. In the reverse denoising process, it corresponds to the maximum number of denoising iterations during sampling.

[0108] In this embodiment, the input to the audio processing model can be a context vector sequence and a current audio frame text vector. The current audio frame text vector is a vector generated by the speech conversion model or music generation model, used to indicate the speech content or lyrics content corresponding to the current audio frame. When the input to the audio processing model is a context vector sequence and a current audio frame text vector, the context vector sequence and the current audio frame text vector serve as conditional vectors to guide the denoising process of random noisy frequency vectors. Specifically, the context vector sequence provides guidance on information such as style, phonemes, accents, and timbre, while the current audio frame text vector provides guidance on text content.

[0109] In this embodiment, the input to the audio processing model can also be a sequence of context vectors. This sequence of context vectors serves as a conditional vector, guiding the denoising process of random, noisy frequency vectors. Based on the context vector sequence, the audio processing model predicts noise and performs denoising.

[0110] In this embodiment, by using an audio processing model based on a diffusion model, at least the context vector sequence is used as a condition vector to guide the denoising process of random noisy frequency vectors, clear and audible audio frames can be obtained directly. At the same time, this embodiment processes the audio directly on a continuous audio sequence, enabling streaming audio generation and improving the fluency, continuity and audibility of the audio.

[0111] S130. Add the current audio frame to the end of the historical audio sequence, and repeat the step of determining the historical audio sequence or the text vector of the historical audio sequence and the current audio frame until the processing of all audio frames is completed.

[0112] In this embodiment, each time a new current audio frame is generated by the audio processing model, it is added to the end of the historical audio sequence to update the historical audio sequence. Since the context vector sequence contains inter-frame temporal dependency information and global information, it can guide the denoising process to obtain audio frames. Therefore, the process of generating new current audio frames is repeated until all audio frames have been processed.

[0113] For example, when the historical audio sequence includes frames 1 through i, it is converted into a context vector sequence. Based on S110-S120, frame (i+1) is generated and added to the end of the historical audio sequence, updating the historical audio sequence to frames 1 through (i+1). Frames 1 through (i+1) are then converted into a context vector sequence, and based on S110-S120, frame (i+2) is generated and added to the end of the historical audio sequence, updating the historical audio sequence to frames 1 through (i+2), and so on.

[0114] The technical solution of this invention involves performing autoregressive processing on a historical audio sequence composed of currently generated historical audio frames to obtain a context vector sequence. This context vector sequence, or a combination of the context vector sequence and the text vector of the current audio frame, is used as input to an audio processing model. The model uses the input data as a condition vector to denoise the random, noisy frequency vectors to be processed, resulting in the current audio frame. This current audio frame is then inserted at the end of the historical audio sequence and used again as the historical audio sequence for generating the next audio frame. This process is repeated until all audio frames are generated. This technical solution directly performs processing operations on the audio frames, avoiding audio encoding and decoding operations and reducing audio processing overhead. It also enables processing based on streaming audio data, achieving streaming audio generation frame by frame, improving the real-time performance of audio generation. Simultaneously, it ensures the semantic coherence of the audio, enhancing the listening experience.

[0115] Example 2:

[0116] Figure 2 This is a flowchart of another audio generation method based on an audio processing model provided in Embodiment 2 of the present invention. Based on the above embodiments, the present invention further specifies the process of obtaining the current audio frame based on the diffusion model and masking mechanism.

[0117] like Figure 2 As shown, the method includes:

[0118] S210. Determine the historical audio sequence or the historical audio sequence and the current audio frame text vector, and perform autoregressive processing on the historical audio sequence to obtain the context vector sequence.

[0119] The process of determining the historical audio sequence, or determining the historical audio sequence and the text vector of the current audio frame, and the process of obtaining the context vector sequence have been described in the above embodiments, and will not be repeated here.

[0120] S220. Using a diffusion model and masking mechanism, based on the context vector sequence or the context vector sequence and the current audio frame text vector, the random noisy frequency vector to be processed is denoised to obtain the current audio frame.

[0121] This embodiment uses a diffusion model based on a masking mechanism as an example to illustrate the audio processing model.

[0122] In the diffusion model, when denoising the random noisy frequency vector to be processed, the denoising range is limited to the current audio frame through a masking mechanism. That is, when generating the i-th audio frame, all audio frames after the i-th audio frame are masked through a masking mechanism. The unmasked part, that is, the part within the current audio frame (frames 1 to (i-1)), is used as a condition vector to guide the denoising process of the i-th audio frame.

[0123] Furthermore, the training steps for the diffusion model can include:

[0124] T1. Obtain the sample audio vector sequence;

[0125] T2. Determine the position to be predicted in the sample audio vector sequence, add a mask to the position to be predicted and the sample audio vectors after the position to be predicted, and obtain the masked sample audio vector sequence.

[0126] T3. Add a preset time step noise to each sample audio vector in the masked sample audio vector sequence to obtain a noisy masked sample audio vector sequence.

[0127] T4. At least input the masked sample audio vector sequence into the preset diffusion model, and let the preset diffusion model perform denoising processing on the masked sample audio vector sequence in parallel;

[0128] T5. Based on the predicted denoising results corresponding to the audio vector sequences of each masked sample and the audio vector sequences of the masked samples before adding noise, determine the model loss and adjust the model parameters based on backpropagation of the model loss.

[0129] T6. Repeat the operation of randomly determining the position to be predicted in the sample audio vector sequence until the preset training conditions are met, and obtain the trained diffusion model.

[0130] This embodiment does not limit the number of sample audio vector sequences, nor the number of sample audio vectors in each sample audio vector sequence.

[0131] Taking a sequence of N sample audio vectors as an example, a position k to be predicted is randomly selected from 1 to N in the sequence to ensure the processing capability of the diffusion model. A mask is added to the positions at and after position k to obscure the sample audio vectors at and after that position. This sequence of sample audio vectors is used as the masked sample audio vector sequence, ensuring that the audio processing model can only see the sample audio vectors before position k during training. Simultaneously, noise is added to the masked sample audio vectors, i.e., the masked sample audio vector sequence, to obtain a noisy masked sample audio vector sequence. Specifically, the added noise can be Gaussian noise; furthermore, noise can be added only to a randomly selected portion of the sample audio vectors in the masked sample audio vector sequence.

[0132] The sample audio vectors corresponding to each position in the masked sample audio vector sequence are denoised in parallel. The predicted noise and the denoised result are obtained at each position. Based on the result and the original sample audio vector before the noise is added at each position, the model loss is calculated, and the model parameters are adjusted by backpropagation.

[0133] In this embodiment, the loss function is established between the real sample audio vector and the predicted denoising result after adding noise. The reason is that in the masked diffusion model, the diffusion model needs to fill the masked position according to the real context. In this case, it is more intuitive for the diffusion model to directly predict the denoised result. This training method can better utilize the surrounding context information and allow the model to learn more directly how to restore the noisy region to real audio frame data.

[0134] The preset training conditions may be that the number of positions to be predicted participating in the model training reaches a preset threshold, or the number of training sessions reaches a preset training limit, or the model performance meets the conditions, such as the prediction accuracy being greater than or equal to a preset accuracy threshold. This embodiment does not limit the specific content of the training conditions.

[0135] Furthermore, since the audio processing model can predict only the current audio frame to achieve streaming processing, during the training of the diffusion model, after adding a mask to the position to be predicted k, the audio vectors of the sample at position k+1 and thereafter can be deleted or masked. That is, the masked sample audio vector sequence at this time only includes the sample audio vector at the position to be predicted k. The advantage of this setting is that it prevents the diffusion model from wasting computational resources on positions that do not need to be predicted during training, thus saving training overhead.

[0136] Furthermore, the first to (k-1) sample audio vectors can be used as the condition vectors of the diffusion model. When the masked sample audio vector sequence is input into the preset diffusion model, the condition vectors can also be set as inputs to the diffusion model. The condition vectors are used to indicate conditional information such as timbre and pitch.

[0137] S230. Add the current audio frame to the end of the historical audio sequence, and repeat the step of determining the historical audio sequence or the text vector of the historical audio sequence and the current audio frame until the processing of all audio frames is completed.

[0138] The technical solution of this embodiment generates a streaming context vector sequence on a continuous audio frame sequence, and performs denoising processing based at least on the context vector sequence using an audio processing model based on a diffusion model and a masking mechanism to predict the current audio frame. This better utilizes contextual information for denoising guidance, directly generating audible audio frames on a continuous audio sequence, achieving streaming audio processing, and improving the real-time performance, semantic coherence, and audio listening experience of audio generation.

[0139] Example 3:

[0140] Figure 3 This is a flowchart of another audio generation method based on an audio processing model provided in Embodiment 3 of the present invention. Based on the above embodiments, the present invention further specifies the process of obtaining the current audio frame based on an autoregressive diffusion model.

[0141] like Figure 3 As shown, the method includes:

[0142] S310. Determine the historical audio sequence or the historical audio sequence and the current audio frame text vector, and perform autoregressive processing on the historical audio sequence to obtain the context vector sequence.

[0143] The process of determining the historical audio sequence, or determining the historical audio sequence and the text vector of the current audio frame, and the process of obtaining the context vector sequence have been described in the above embodiments, and will not be repeated here.

[0144] S320. Using an autoregressive diffusion model, based on the context vector sequence or the context vector sequence and the current audio frame text vector, perform autoregressive denoising processing on the random noisy frequency vector to be processed to obtain the current audio frame.

[0145] This embodiment uses the Autoregressive Diffusion Transformer (AR DiT) audio processing model as an example for illustration.

[0146] The input to the autoregressive diffusion model includes at least the context vector sequence output by the autoregressive sequence processing model. At each time step, the autoregressive diffusion model uses the context vector sequence as a conditional vector to guide the denoising process, thereby removing noise based on contextual information and inter-frame correlations to predict the current audio frame. In autoregressive mode, the length of the context vector sequence continuously increases as the processing is repeated.

[0147] Furthermore, the training steps for the autoregressive diffusion model include:

[0148] D1. Add preset time-step noise to the sample audio vectors to obtain a noisy sample audio vector sequence;

[0149] D2. Determine the position to be predicted in the sample audio vector sequence, input the noisy sample audio vector sequence into a preset diffusion model, and let the preset diffusion model predict the prediction noise corresponding to the position to be predicted based on the causal attention mechanism.

[0150] D3. Based on the predicted noise and the actual noise superimposed on the sample audio vectors, calculate the model loss and adjust the model parameters through backpropagation.

[0151] D4. Repeat the above training steps until the autoregressive diffusion model meets the preset training conditions, and obtain the trained autoregressive diffusion model.

[0152] Similarly, the superimposed noise can be Gaussian noise, with T steps of noise superimposed on the sample audio vector. A position to be predicted is randomly selected; in this embodiment, the position to be predicted is actually the time step to be predicted, with a value ranging from 1 to T. The noisy sample audio vector sequence corresponding to the position to be predicted is input into a pre-set autoregressive diffusion model. The autoregressive diffusion model, based on a causal attention mechanism, predicts the prediction noise at the position to be predicted.

[0153] Based on the predicted noise and the actual noise superimposed on the sample audio vector at the location to be predicted, the model loss is calculated and the model parameters are adjusted by backpropagation.

[0154] In this embodiment, the loss function of the autoregressive diffusion model is established between the predicted noise and the actual superimposed noise. This arrangement provides the model with a relatively consistent training signal across different time steps, making convergence easier, especially in causal modes. This is particularly important in autoregressive diffusion models that predict noise one step at a time, as these models require accurate predictions at each step. If the loss is calculated between the predicted denoising result and the actual result, the autoregressive diffusion model may develop unstable gradients when learning to remove noise at different time steps (with varying noise levels): when noise is low, the model needs fine-tuning; when noise is high, it needs significant adjustments. This makes it difficult for the model to perform both tasks well simultaneously. Therefore, establishing the loss function between the predicted noise and the actual superimposed noise is a more optimized choice. Regardless of the noise level, the model's task is to predict the noise added to the sample audio vector. This task has a more consistent difficulty and nature across different time steps, thus providing a more stable training signal, a more consistent training objective, and improved training efficiency.

[0155] S330. Add the current audio frame to the end of the historical audio sequence, and repeat the step of determining the historical audio sequence or the text vector of the historical audio sequence and the current audio frame until the processing of all audio frames is completed.

[0156] The technical solution of this embodiment generates a streaming context vector sequence on a continuous audio frame sequence, and uses an audio processing model based on an autoregressive diffusion model to use the context vector sequence as a conditional vector to guide the denoising process and predict the current audio frame. This directly generates audible audio frames on a continuous audio sequence, achieving streaming audio processing and improving the real-time performance of audio generation, semantic coherence, and audio listening experience.

[0157] Example 4:

[0158] Figure 4 This is a schematic diagram of an audio generation device based on an audio processing model provided in Embodiment 4 of the present invention. Figure 4 As shown, the device includes:

[0159] The context vector sequence determination module 410 is used to determine a historical audio sequence or a historical audio sequence and the current audio frame text vector, and to perform autoregressive processing on the historical audio sequence to obtain a context vector sequence.

[0160] The historical audio sequence consists of at least two historical audio frames;

[0161] The current audio frame determination module 420 is used by the audio processing model to perform noise reduction processing on the random noisy frequency vector to be processed based on the context vector sequence or the context vector sequence and the current audio frame text vector to obtain the current audio frame;

[0162] The audio sequence update module 430 is used to add the current audio frame to the end of the historical audio sequence and repeatedly execute the step of determining the historical audio sequence or the text vector of the historical audio sequence and the current audio frame until the processing of all audio frames is completed.

[0163] Based on the above embodiments, optionally, the context vector sequence determination module 410 includes:

[0164] The target audio frame conversion unit is used to convert the target audio frame into a target floating-point vector through an autoregressive sequence processing model.

[0165] A position encoding addition unit is used to add position encoding to a target floating-point vector, wherein the position encoding is used to represent the position of the target audio frame in a historical audio sequence;

[0166] The association information determination unit is used to determine the association information between the target floating-point vector and the historical floating-point vectors corresponding to the historical audio frames preceding the target audio frame, based on a causal attention mechanism.

[0167] The global context vector determination unit is used to perform information fusion based on associated information to obtain a global context vector that matches the target audio frame;

[0168] The context vector sequence determination unit is used to obtain the context vector sequence based on the global context vector of each audio frame.

[0169] Based on the above embodiments, optionally, the current audio frame determination module 420 includes:

[0170] The diffusion model processing unit based on the masking mechanism is used to perform denoising processing on the random noisy frequency vector to be processed based on the context vector sequence or the context vector sequence and the current audio frame text vector through the diffusion model and the masking mechanism, so as to obtain the current audio frame.

[0171] In the diffusion model, when denoising the random noisy frequency vector to be processed, the denoising range is limited to the current audio frame through a masking mechanism.

[0172] Optionally, based on the above embodiments, the apparatus further includes:

[0173] The first sample audio vector sequence determination module is used to obtain the sample audio vector sequence;

[0174] The masking module is used to determine the position to be predicted in the sample audio vector sequence, and to add a mask to the position to be predicted and the sample audio vectors after the position to be predicted, so as to obtain the masked sample audio vector sequence.

[0175] The noise superposition module is used to superimpose noise at a preset time step onto each sample audio vector in the masked sample audio vector sequence to obtain a noisy masked sample audio vector sequence.

[0176] The denoising module is used to input at least the masked sample audio vector sequence into a preset diffusion model, and to enable the preset diffusion model to perform denoising processing on the masked sample audio vector sequence in parallel.

[0177] The first model parameter adjustment module is used to determine the model loss based on the prediction denoising results corresponding to each masked sample audio vector sequence and the masked sample audio vector sequence before adding noise, and adjust the model parameters based on the backpropagation of the model loss.

[0178] The diffusion model determination module is used to repeatedly and randomly execute the operation of determining the position to be predicted in the sample audio vector sequence until the preset training conditions are met, and the trained diffusion model is obtained.

[0179] Based on the above embodiments, optionally, the current audio frame determination module 420 includes:

[0180] The autoregressive diffusion model processing module is used to perform autoregressive denoising processing on the random noisy frequency vector to be processed based on the context vector sequence or the context vector sequence and the current audio frame text vector, so as to obtain the current audio frame.

[0181] Optionally, based on the above embodiments, the apparatus further includes:

[0182] The second sample audio vector sequence determination module is used to superimpose preset time step noise on the sample audio vectors to obtain a noisy sample audio vector sequence.

[0183] The prediction noise determination module is used to determine the position to be predicted in the sample audio vector sequence. The noisy sample audio vector sequence is input into a preset diffusion model, and the preset diffusion model is used to predict the prediction noise corresponding to the position to be predicted based on a causal attention mechanism.

[0184] The second model parameter adjustment module is used to calculate the model loss based on the predicted noise and the actual noise superimposed on the sample audio vectors, and adjust the model parameters through backpropagation.

[0185] The autoregressive diffusion model determination module is used to repeatedly execute the above training steps until the autoregressive diffusion model meets the preset training conditions, thus obtaining the trained autoregressive diffusion model.

[0186] Based on the above embodiments, optionally, before the current audio frame determination module 420 performs processing, the context vector sequence determination module 410 further includes:

[0187] The speech conversion unit is used to convert the text to be processed into speech using a speech conversion model to obtain a historical audio sequence, or a historical audio sequence and a text vector of the current audio frame. The text to be processed includes a textual description of the audio frame and / or speech content.

[0188] The music generation unit is used to process the text to be processed through a music generation model to obtain a historical audio sequence, or a historical audio sequence and a text vector of the current audio frame. The text to be processed includes a textual description of the audio frame and / or lyrics.

[0189] Based on the above embodiments, optionally, the speech conversion unit is specifically used for:

[0190] If the text to be processed includes textual descriptions, then the textual descriptions are converted into speech conditional vectors, and at least one text vector sequence is generated based on the text to be processed using a text encoder, or an artificial intelligence model and a text encoder.

[0191] Obtain an autoregressive acoustic model, input the text vector sequence into the autoregressive acoustic model, and let the autoregressive acoustic model generate historical audio sequences, or historical audio sequences and current audio frame text vectors, step by step based on the text vector sequence.

[0192] Alternatively, the text vector sequence and the speech condition vector can be input into the autoregressive acoustic model, causing the autoregressive acoustic model to autoregressively generate historical audio sequences, or historical audio sequences and current audio frame text vectors, based on the text vector sequence and the condition vector.

[0193] Based on the above embodiments, optionally, the music generation unit is specifically used for:

[0194] If the text to be processed includes textual descriptions, then based on an artificial intelligence model, the textual descriptions are converted into music conditional vectors, and based on a text encoder, or an artificial intelligence model and a text encoder, at least one music vector sequence is generated based on the text to be processed.

[0195] Obtain an autoregressive music generation model, input the music vector sequence into the autoregressive music generation model, and let the autoregressive music generation model generate historical audio sequences or historical audio sequences and current audio frame text vectors step by step based on the music vector sequence.

[0196] Alternatively, the music vector sequence and the music condition vector can be input into the autoregressive music generation model, so that the autoregressive music generation model can generate historical audio sequences, or historical audio sequences and current audio frame text vectors, step by step based on the music vector sequence and the music condition vector.

[0197] The historical audio sequence includes at least two channels, and audio frames at the same position in at least two channels are generated in parallel.

[0198] The audio generation device based on the audio processing model provided in the embodiments of the present invention can execute the audio generation method based on the audio processing model provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0199] Example 5:

[0200] Figure 5 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0201] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0202] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0203] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as an audio generation method based on an audio processing model.

[0204] In some embodiments, the audio generation method based on an audio processing model can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the audio generation method based on an audio processing model described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the audio generation method based on an audio processing model by any other suitable means (e.g., by means of firmware).

[0205] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0206] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable audio generation apparatus based on an audio processing model, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0207] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0208] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0209] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0210] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0211] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0212] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An audio generation method based on an audio processing model, characterized by, The method comprises the following steps: determining a history audio sequence or a history audio sequence and a current audio frame text vector, performing autoregressive processing on the history audio sequence to obtain a context vector sequence, comprising: converting a target audio frame into a target floating point vector through an autoregressive sequence processing model; adding a position encoding to the target floating point vector, wherein the position encoding is used to represent the position of the target audio frame in the history audio sequence; determining the association information between the target floating point vector and the history floating point vector corresponding to the history audio frame before the target audio frame based on a causal attention mechanism; performing information fusion based on the association information to obtain a global context vector matching the target audio frame; obtaining a context vector sequence according to the global context vector of each audio frame; wherein the history audio sequence is composed of at least two history audio frames; when determining the history audio sequence, the audio processing model performs denoising processing on a random noisy audio vector to be processed based on the context vector sequence to obtain a current audio frame; when determining the history audio sequence and the current audio frame text vector, the audio processing model performs denoising processing on a random noisy audio vector to be processed based on the context vector sequence and the current audio frame text vector to obtain a current audio frame; adding the current audio frame to the end of the history audio sequence, and repeating the steps of determining the history audio sequence or the history audio sequence and the current audio frame text vector until the processing of all audio frames is completed.

2. The method of claim 1, wherein, The audio processing model performs denoising processing on a random noisy audio vector to be processed based on the context vector sequence to obtain a current audio frame, comprising: performing denoising processing on a random noisy audio vector to be processed based on the context vector sequence through a diffusion model and a mask mechanism to obtain a current audio frame; The audio processing model performs denoising processing on a random noisy audio vector to be processed based on the context vector sequence and the current audio frame text vector to obtain a current audio frame, comprising: performing denoising processing on a random noisy audio vector to be processed based on the context vector sequence and the current audio frame text vector through a diffusion model and a mask mechanism to obtain a current audio frame; wherein the diffusion model limits the denoising range to the current audio frame through the mask mechanism when performing denoising on the random noisy audio vector to be processed.

3. The method of claim 2, wherein, The training process of the diffusion model comprises: obtaining a sample audio vector sequence; determining a to-be-predicted position of the sample audio vector sequence, adding a mask to the to-be-predicted position and the sample audio vector after the to-be-predicted position to obtain a masked sample audio vector sequence; adding a preset time step noise to each sample audio vector in the masked sample audio vector sequence to obtain a noisy masked sample audio vector sequence; inputting at least the masked sample audio vector sequence into a preset diffusion model, and enabling the preset diffusion model to perform denoising processing on the masked sample audio vector sequence in parallel; determining a model loss based on the predicted denoising processing result corresponding to each masked sample audio vector sequence and the masked sample audio vector sequence before the noise is added, and adjusting the model parameters based on the model loss. The operation of determining the to-be-predicted position of the sample audio vector sequence is repeatedly performed randomly until a preset training condition is met, and a trained diffusion model is obtained.

4. The method of claim 1, wherein, The audio processing model performs denoising processing on the to-be-processed random noisy audio vector based on the context vector sequence, to obtain a current audio frame, and further comprises: The audio processing model performs denoising processing on the to-be-processed random noisy audio vector based on the context vector sequence and the current audio frame text vector, to obtain a current audio frame, and further comprises: The audio processing model performs denoising processing on the to-be-processed random noisy audio vector based on the context vector sequence and the current audio frame text vector, to obtain a current audio frame, and further comprises: The audio processing model performs denoising processing on the to-be-processed random noisy audio vector based on the context vector sequence and the current audio frame text vector, to obtain a current audio frame, and further comprises:

5. The method of claim 4, wherein, The training step of the autoregressive diffusion model comprises: A preset time step noise is superimposed on the sample audio vector to obtain a noisy sample audio vector sequence; A to-be-predicted position of the sample audio vector sequence is determined, and the noisy sample audio vector sequence is input into a preset diffusion model, so that the preset diffusion model predicts a predicted noise corresponding to the to-be-predicted position based on a causal attention mechanism; Based on the predicted noise and the actual noise superimposed on the sample audio vector, a model loss is calculated, and the model parameters are adjusted through back propagation; The above training steps are repeatedly performed until the autoregressive diffusion model meets a preset training condition, and a trained autoregressive diffusion model is obtained.

6. The method of claim 1, wherein, Before the audio processing model performs audio processing, a historical audio sequence or a historical audio sequence and a current audio frame text vector are determined, comprising: A speech conversion model is used to convert the to-be-processed text into a historical audio sequence or a historical audio sequence and a current audio frame text vector, wherein the to-be-processed text comprises a textual description of an audio frame and / or speech content; Alternatively, a music generation model is used to process the to-be-processed text to obtain a historical audio sequence or a historical audio sequence and a current audio frame text vector, wherein the to-be-processed text comprises a textual description of an audio frame and / or lyrics content.

7. The method of claim 6, wherein, A speech conversion model is used to convert the to-be-processed text into a historical audio sequence or a historical audio sequence and a current audio frame text vector, comprising: If the to-be-processed text comprises a textual description, the textual description is converted into a speech condition vector, and at least one text vector sequence is generated based on the to-be-processed text based on a text encoder or an artificial intelligence model and a text encoder; An autoregressive acoustic model is obtained, and the text vector sequence is input into the autoregressive acoustic model, so that the autoregressive acoustic model generates a historical audio sequence or a historical audio sequence and a current audio frame text vector step by step in sequence based on the text vector sequence; or, inputting the text vector sequence and the speech condition vector into the autoregressive acoustic model, and enabling the autoregressive acoustic model to generate a historical audio sequence or a historical audio sequence and a current audio frame text vector step by step in sequence based on the text vector sequence and the condition vector.

8. The method of claim 6, wherein, The music generation model is used to process the to-be-processed text to obtain a historical audio sequence or a historical audio sequence and a current audio frame text vector, including: If the to-be-processed text includes a literal description, the literal description is converted into a music condition vector based on an artificial intelligence model, and at least one music vector sequence is generated based on a text encoder or an artificial intelligence model and a text encoder based on the to-be-processed text; An autoregressive music generation model is obtained, the music vector sequence is input into the autoregressive music generation model, and the autoregressive music generation model is enabled to generate a historical audio sequence or a historical audio sequence and a current audio frame text vector step by step in sequence based on the music vector sequence; or, inputting the music vector sequence and the music condition vector into the autoregressive music generation model, and enabling the autoregressive music generation model to generate a historical audio sequence or a historical audio sequence and a current audio frame text vector step by step in sequence based on the music vector sequence and the music condition vector; The historical audio sequence includes at least two channels, and audio frames at the same position in the at least two channels are generated in parallel.

9. An audio generation apparatus based on an audio processing model, the apparatus comprising: It includes: A context vector sequence determination module is configured to determine a historical audio sequence or a historical audio sequence and a current audio frame text vector, and perform autoregressive processing on the historical audio sequence to obtain a context vector sequence. The context vector sequence determination module includes: A target audio frame conversion unit is configured to convert a target audio frame into a target floating point vector through an autoregressive sequence processing model; A position encoding addition unit is configured to add position encoding to the target floating point vector, and the position encoding is used to represent the position of the target audio frame in the historical audio sequence; An association information determination unit is configured to determine association information between the target floating point vector and a historical floating point vector corresponding to a historical audio frame before the target audio frame based on a causal attention mechanism; A global context vector determination unit is configured to perform information fusion based on the association information to obtain a global context vector matching the target audio frame; A context vector sequence determination unit is configured to obtain a context vector sequence according to the global context vector of each audio frame. The historical audio sequence is composed of at least two historical audio frames. A current audio frame determination module is configured to, when determining the historical audio sequence, perform denoising processing on a to-be-processed random noisy audio vector based on the context vector sequence to obtain a current audio frame, and when determining the historical audio sequence and the current audio frame text vector, perform denoising processing on a to-be-processed random noisy audio vector based on the context vector sequence and the current audio frame text vector to obtain a current audio frame. an audio sequence updating module for adding the current audio frame to the end of the history audio sequence and repeating the steps of determining the text vector of the history audio sequence or the history audio sequence and the current audio frame until the processing of all audio frames is completed.

Images