Method and device for training neural network model and method and device for generating audio

Abstract

The embodiment of the invention provides a method and device for training a neural network model and a method and device for generating audio. The method for training the neural network model comprises the steps that a training sample including audio and text description is acquired, and the text description is used for describing sound effects contained in the audio and sound effect proceeding time; tense information is extracted based on the text description, and the tense information comprises description information for the sound effect and timestamp information of the sound effect; encoding the audio, the text description and the tense information to obtain audio features, text features and tense features; performing noise addition on the audio features to obtain noise-added features; based on the text feature and the tense feature, denoising the noise adding feature by using a diffusion model to obtain a denoised feature; and for each sample, training a diffusion model based on the difference between the audio feature and the de-noised feature. The trained diffusion model can be used for accurately completing an audio generation task.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and more specifically, to a method, apparatus, computer program product, and storage medium for training a neural network model, and a method, apparatus, computer program product, and storage medium for generating audio. Background Art

[0002] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0003] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0004] Audio technology is a key branch of artificial intelligence and a key enabler for human-computer interaction. It encompasses the analysis, processing, and application of speech signals, primarily used in speech recognition, audio generation, and speech perception. Audio technology can digitize speech signals, extract speech features, and perform classification and recognition. It can also convert text into speech signals and synthesize them.

[0005] With the continuous development of artificial intelligence (AI), the application scenarios of audio technology are also expanding. Audio technology has a wide range of applications, including voice assistants, smart homes, intelligent customer service, and voice search. Voice processing technology allows people to interact with machines more conveniently, improving the experience and efficiency of human-computer interaction. For example, audio generation technology can be used to generate audio for sound effects and human-computer interaction.

[0006] At present, as users have increasingly higher requirements for audio generation, how to more accurately generate the audio required by users is one of the key research directions in this field. Summary of the Invention

[0007] In order to utilize artificial intelligence technology to more accurately complete speech synthesis tasks, the present disclosure provides a method for training a neural network model, comprising: obtaining training samples, wherein the training samples include: audio and text description, wherein the text description is used to describe the sound effects contained in the audio and the time when the sound effects are performed; extracting temporal information based on the text description, wherein the temporal information includes: description information for the sound effects and timestamp information of the sound effects; encoding the audio, the text description and the temporal information respectively to obtain audio features, text features and temporal features; denoising the audio features to obtain noisy features; denoising the noisy features using a diffusion model based on the text features and the temporal features to obtain denoised features; and for each of the samples, training the diffusion model based on the difference between the audio features and the denoised features.

[0008] According to an embodiment of the present disclosure, obtaining a training sample includes: obtaining an original sample, the original sample including: original audio and original text description, wherein the original text description is used to describe the original sound effects contained in the original audio and the time when the original sound effects are performed; and combining different original samples to obtain a combined sample, and using the combined sample as the training sample, wherein combining different original samples includes: combining the original audios of the different original samples, and correspondingly combining the original text descriptions of the different original samples.

[0009] According to an embodiment of the present disclosure, adding noise to the audio feature to obtain the noise-added feature includes: determining a random noise feature; and randomly determining a combination ratio of the audio feature and the random noise feature to generate the noise-added feature.

[0010] According to an embodiment of the present disclosure, the diffusion model includes multiple attention networks, and denoising the noisy features using the diffusion model includes: denoising the noisy features sequentially through the multiple attention networks.

[0011] An embodiment of the present disclosure also provides a method for generating audio, comprising: obtaining an indication for generating audio, and determining a text description for the audio to be generated based on the indication, wherein the text description is used to describe the sound effects contained in the audio to be generated, and the time when the sound effects are performed; extracting temporal information based on the text description, wherein the temporal information includes: description information for the sound effects, and timestamp information of the sound effects; encoding the text description and the temporal information respectively to obtain text features and temporal features; based on random noise features, the text features and the temporal features, processing the random noise features using a diffusion model to obtain audio features; and decoding the audio features to obtain the audio to be generated.

[0012] According to an embodiment of the present disclosure, processing the random noise feature using a diffusion model to obtain an audio feature includes: denoising the random noise feature using the diffusion model to obtain a first denoised feature, and denoising the first denoised feature to obtain a first noisy feature; denoising the nth noisy feature using the diffusion model to obtain an n+1th denoised feature, and denoising the n+1th denoised feature to obtain an n+1th noisy feature; and denoising the Nth noisy feature using the diffusion model to obtain an N+1th denoised feature, and using the N+1th denoised feature as the audio feature, wherein 1≤n<N.

[0013] According to an embodiment of the present disclosure, determining a text description for the audio to be generated based on the indication includes: determining a scene description for the scene reflected by the audio based on the indication; and generating the text description based on the scene description.

[0014] An embodiment of the present disclosure also provides an apparatus for training a neural network model, comprising: a sample acquisition module, configured to acquire training samples, wherein the training samples include audio and text descriptions, wherein the text descriptions are used to describe the sound effects contained in the audio and the time when the sound effects occur; an information extraction module, configured to extract temporal information based on the text descriptions, wherein the temporal information includes description information for the sound effects and timestamp information of the sound effects; an encoding module, configured to encode the audio, the text descriptions and the temporal information respectively to obtain audio features, text features and temporal features; a noise addition module, configured to add noise to the audio features to obtain noisy features; a denoising module, configured to denoise the noisy features using a diffusion model based on the text features and the temporal features to obtain denoised features; and a training module, configured to train the diffusion model for each of the samples based on the difference between the audio features and the denoised features.

[0015] An embodiment of the present disclosure also provides a device for generating audio, including: an acquisition module, configured to: obtain an indication of generating audio, and determine a text description of the audio to be generated based on the indication, wherein the text description is used to describe the sound effects contained in the audio to be generated, and the time when the sound effects are performed; an information extraction module, configured to: extract temporal information based on the text description, wherein the temporal information includes: description information for the sound effects, and timestamp information of the sound effects; an encoding module, configured to: encode the text description and the temporal information respectively to obtain text features and temporal features; a feature processing module, configured to: process the random noise features using a diffusion model based on random noise features, the text features and the temporal features to obtain audio features; and a decoding module, configured to: decode the audio features to obtain the audio to be generated.

[0016] An embodiment of the present disclosure further provides a computer program product, which includes computer software code. When the computer software code is executed by a processor, it provides the above method.

[0017] An embodiment of the present disclosure further provides a computer-readable storage medium having computer-executable instructions stored thereon, which provide the above method when executed by a processor.

[0018] The disclosed method extracts temporal information from the textual description corresponding to the audio to be generated, without requiring additional user input of temporal information. The audio is then generated based on both the textual description and the temporal information, effectively improving the accuracy of the generated audio, particularly in terms of temporal detail. Furthermore, the disclosed method can generate rich audio from simple scene descriptions, without requiring the text used to generate the audio to contain precise descriptions of the sound effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] To more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some exemplary embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0020] Herein, in the accompanying drawings:

[0021] Figure 1 A schematic diagram showing an application scenario according to an embodiment of the present disclosure is shown;

[0022] Figure 2 is an example schematic diagram illustrating a scenario of performing audio generation based on a neural network model for generating audio according to an embodiment of the present disclosure;

[0023] Figure 3 is a schematic diagram illustrating a training process of a neural network model for generating audio according to an embodiment of the present disclosure;

[0024] Figure 4A is a schematic flow chart illustrating a method for training a neural network model according to an embodiment of the present disclosure;

[0025] Figure 4B-4D is a schematic diagram illustrating a processing procedure based on a diffusion model according to an embodiment of the present disclosure;

[0026] Figure 5 is a schematic diagram illustrating an audio generation process according to an embodiment of the present disclosure;

[0027] Figure 6 is a schematic flow chart illustrating a method for generating audio according to an embodiment of the present disclosure;

[0028] Figure 7 1 is a schematic diagram showing the composition of an apparatus for training a neural network model according to an embodiment of the present disclosure;

[0029] Figure 8 is a schematic diagram showing the composition of an apparatus for generating audio according to an embodiment of the present disclosure; and

[0030] Figure 9 2 is a diagram illustrating an architecture of a computing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present disclosure more apparent, the exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0032] Furthermore, in this specification and the drawings, steps and elements having substantially the same or similar features are denoted by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted.

[0033] In addition, in this specification and the drawings, elements are described in singular or plural form, depending on the embodiment. However, the singular and plural forms are appropriately selected for the situations presented merely for convenience of explanation and are not intended to limit the present disclosure thereto. Therefore, the singular form may include the plural form, and the plural form may also include the singular form, unless the context clearly indicates otherwise.

[0034] In this specification and the accompanying drawings, substantially the same or similar steps or elements are denoted by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted. At the same time, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance or ranking.

[0035] To facilitate description of the present disclosure, concepts related to the present disclosure are introduced below.

[0036] Key technologies in speech technology include automatic speech recognition (ASR), text-to-speech (TTS), and voiceprint recognition. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech becoming one of the most promising methods of human-computer interaction. Large model technology is revolutionizing the development of speech technology. Pre-trained models such as WavLM and UniSpeech, which leverage the Transformer architecture, possess strong generalization and versatility, enabling them to effectively handle a wide range of speech processing tasks.

[0037] The development of artificial intelligence (AI) has also revolutionized speech technology. For example, AI-based speech recognition technology can convert human speech into text. This advancement has made speech a more direct and convenient means of human-computer interaction, such as voice assistants and voice commands. AI has also improved speech synthesis technology, enabling computers to generate high-quality, natural, and fluent speech. Applications of this technology include sound effect synthesis, voice assistants, audiobooks, and automated voice navigation.

[0038] Neural networks (NNs), a key branch of artificial intelligence, are network structures that mimic the behavioral characteristics of animal neural networks to process information. Neural networks are composed of a large number of interconnected nodes (or neurons), which process information by learning and training input information based on specific computational models. A neural network consists of an input layer, a hidden layer, and an output layer. The input layer receives input signals, while the output layer outputs the neural network's computational results. The hidden layer handles computational processes such as learning and training and serves as the network's memory unit. The hidden layer's memory function is represented by a weight matrix, with each neuron typically assigned a weight coefficient.

[0039] The diffusion model is a deep generative model that can predict original data based on noisy data. The training process of the diffusion model includes a forward process (or diffusion process) and a reverse process (or inverse diffusion process). In the forward process, noise is added to the original data to obtain noisy data. In the reverse process, the predicted original data is recovered from the noisy data. The diffusion model can be trained by comparing the differences between the original data and the predicted original data. Through this training process, the trained diffusion model has the ability to predict the original data based on noisy data. The principle of the diffusion model is relatively simple and has good performance and diversity in generative tasks.

[0040] The various neural networks (or neural network models) that can be used in the embodiments of the present disclosure below can all be artificial intelligence models, especially artificial intelligence-based neural network models. Typically, artificial intelligence-based neural network models are implemented as acyclic graphs in which neurons are arranged in different layers. Typically, a neural network model includes an input layer and an output layer, which are separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation that is useful for generating output in the output layer. The network nodes (i.e., neurons) are fully connected to the nodes in the adjacent layers via edges, and there are no edges between the nodes in each layer. The data received at the nodes of the input layer of the neural network are propagated to the nodes of the output layer via any one of the hidden layer, activation layer, pooling layer, convolutional layer, etc. The input and output of the neural network model can take various forms, and the present disclosure is not limited thereto.

[0041] In summary, the present disclosure relates to technologies such as artificial intelligence and speech synthesis. The following further describes embodiments of the present disclosure in conjunction with the accompanying drawings.

[0042] First refer to Figure 1 Describe the application scenarios of the method and corresponding device according to the embodiments of the present disclosure. Figure 1 A schematic diagram of an application scenario 100 according to an embodiment of the present disclosure is shown, wherein a server 110 and multiple terminals 120 are schematically shown.

[0043] The neural network model for generating audio according to the embodiment of the present disclosure can be integrated into an apparatus for generating audio and located in various electronic devices, for example, Figure 1 The server 110 and any electronic device in the multiple terminals 120. For example, the neural network model for generating audio can be integrated in the terminal 120. The terminal 120 includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, and car terminals. For another example, the neural network model for generating audio can also be integrated in the server 110. The server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in this disclosure.

[0044] It is understood that the apparatus for generating audio using the embodiments of the present disclosure may be a terminal, a server, or a system consisting of a terminal and a server. The method for generating audio using the embodiments of the present disclosure may be executed on a terminal, a server, or both.

[0045] The neural network model for generating audio provided by the embodiments of the present disclosure can be used to perform various text-to-audio synthesis (T2A) tasks, such as movie sound creation, virtual reality sound effects, game sound effects, audio editing, character dubbing, audio synthesis of interactive systems (e.g., navigation systems, intelligent recommendation systems, etc.), and the like.

[0046] The neural network model for generating audio provided by the embodiments of the present disclosure may also involve artificial intelligence cloud services in the field of cloud technology. Among them, cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network in a wide area network or a local area network to realize the calculation, storage, processing and sharing of data. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the application of cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of technical network systems require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0047] Among them, AI cloud services, also commonly referred to as AIaaS (AI as a Service), are a mainstream AI platform service model. Specifically, AIaaS platforms separate several common AI services and provide them as standalone or packaged services in the cloud. This service model is similar to opening an AI-themed mall: all developers can access one or more AI services provided by the platform through the application programming interface (API). Some experienced developers can also use the platform's AI framework and AI infrastructure to deploy and operate their own cloud AI services.

[0048] Figure 2 2 is an example schematic diagram illustrating a scenario 200 in which audio generation is performed based on a neural network model for generating audio according to an embodiment of the present disclosure.

[0049] During the training phase, server 110 may train a neural network model for generating audio based on training samples, where the training samples include audio and text descriptions, where the text descriptions describe the sound effects contained in the audio and the time at which the sound effects occur. After training is complete, the server may deploy the trained neural network model for generating audio to one or more servers (or cloud services) to provide artificial intelligence services related to generating audio.

[0050] It is worth noting that all audio or sound effects used in this disclosure comply with the legality, ethics, and privacy requirements of laws and regulations. Specifically, all audio or sound effects are sourced from legitimate sources and have been collected with the user's explicit permission. Furthermore, all audio or sound effects used in this disclosure adhere to privacy protection principles, have been rigorously screened and cleaned, and will not be disclosed to any unauthorized third party.

[0051] In the stage of audio generation based on the neural network model for generating audio, it is assumed that a client or application that interacts with the server 110 for generating audio (i.e., various clients or applications for generating audio (e.g., scene sound effect generation, scene dubbing, etc.)) has been installed on the user terminal 120 for generating audio. The user terminal 120 can send a request for audio generation to the server 110 corresponding to the application through the network to request the neural network for generating audio deployed on the server 110 to generate audio. For example, after the server 110 receives the request for audio generation, it uses the trained neural network model for generating audio to respond to the request for audio generation, and feeds back the audio generation result (i.e., the audio required to be generated by the user) to the user terminal 120. The user terminal 120 can receive the audio generation result. Afterwards, the user terminal 120 can perform further processing (e.g., dubbing, audio editing, etc.) based on the audio generation result.

[0052] It is worth noting that Figure 2 The training sample set shown in can also be set in advance. For example, refer to Figure 2 The server can obtain training data (e.g., audio and text description, where the text description is used to describe the sound effects contained in the audio and the time when the sound effects occur) from the database 130, and then determine a training sample set for the neural network model used to generate the audio.

[0053] Optionally, existing samples (each sample in the audio data set includes audio, sound effects contained in the audio, and a text description of the time at which the sound effects are performed) can be mixed to expand the training samples, making the training sample set richer and more flexible without being restricted by the existing sample set.

[0054] Current text-to-audio synthesis tasks typically use large language models to extract semantic information to generate audio that is semantically relevant to the text. However, this approach only focuses on the content information of the text and lacks the fine-grained temporal information used for audio synthesis tasks, resulting in the generated audio often being inconsistent with the expected temporal order of the text. For example, for a description like "Please generate sounds A and B that play simultaneously, with each sound lasting 1-3 seconds", the model may only focus on sounds A and B, resulting in the generated audio not meeting the time requirement of each sound lasting 1-3 seconds, or the two sounds being synthesized sequentially rather than simultaneously (for example, 1 second of sound A followed by 1 second of sound B, rather than sounds A and B playing simultaneously).

[0055] Therefore, in order to more accurately control the time sequence of each sound effect in the audio generation process, the present disclosure proposes a new neural network model for generating audio. Figure 3-Figure 8 The principle of the neural network model for generating audio is further explained.

[0056] Figure 3 is a schematic diagram illustrating a training process of a neural network model for generating audio according to an embodiment of the present disclosure.

[0057] like Figure 3 As shown, during the training process, the present disclosure encodes the audio, the text description corresponding to the audio, and the temporal information corresponding to the text description respectively to obtain audio features, text features and temporal features, wherein the text description is used to describe the sound effects contained in the audio and the time when the sound effects are performed. Then, the audio features obtained after encoding are denoised to obtain noisy features (i.e., the forward process in the diffusion model training). The text features and temporal features are used as guiding conditions for the diffusion model input, and the diffusion model can denoise the noisy features to obtain denoised features (i.e., the reverse process in the diffusion model training). The diffusion model can be trained by comparing the difference between the original audio features and the denoised features. During each training iteration, noise can be randomly added to the audio features. After multiple trainings, the trained diffusion model can obtain predicted audio features based on the noise features, text features and temporal features, and then decode the audio corresponding to the text description based on the predicted audio features.

[0058] Specifically, for the aforementioned training process, the training samples used for model training can include the following information: audio and a text description corresponding to the audio, where the text description is used to describe the sound effects contained in the audio and the time when the sound effects occurred. To improve the granularity of temporal information processing, temporal information can be further extracted based on the text description, where the temporal information includes: description information for the sound effect and the timestamp information of the sound effect.

[0059] For example, an audio clip may consist of a dog barking for 3 seconds followed by a cat meowing for 2 seconds. The corresponding text description may be "3 seconds of dog barking, 2 seconds of cat meowing." The temporal information extracted based on the text description may be "<dog barking><start 0 seconds><end 3 seconds>, ​​<cat meowing><start 3 seconds><end 5 seconds>." Alternatively, the temporal information extracted based on the text description may be obtained by analyzing the text description using a language model. The temporal information of all samples may have the same format for easier processing.

[0060] Alternatively, to expand the training sample set, existing training samples can be combined to obtain richer training samples. For example, the original training sample set may only contain samples of "3 seconds of dog barking" and "2 seconds of cat meowing." By combining them, we can obtain samples of "3 seconds of dog barking, then 2 seconds of cat meowing," "1-3 seconds of dog barking, then 2-3 seconds of cat meowing," and so on.

[0061] like Figure 3 As shown, the temporal information can be encoded by the temporal encoder to obtain the temporal feature C time The text feature C can be obtained by encoding the text description through the text encoder desc For the input audio, the Mel feature extraction can be performed on the audio to obtain the Mel spectrum, and then the Mel spectrum is encoded using an audio encoder (for example, a variational auto-encoder (VAE)) to obtain the audio feature C audio , then, the audio feature C audio Add noise feature C noise The noise feature C can be obtained mix According to an embodiment of the present disclosure, C mix It can be calculated by the following formula (1):

[0062]

[0063] Where t is the index number, is a predetermined coefficient corresponding to index number t, and satisfies When training the model, you can randomly select t and the corresponding As t increases, The value can be gradually reduced.

[0064] Diffusion model is used to calculate the noise-based feature C mix To predict the denoising feature C audio '. The input of the diffusion model is the noise feature C mix , temporal feature Ctime , and text features C desc , the output is the predicted denoising feature C audio ', where temporal feature C time and text features C desc As a diffusion model based on the noise feature C mix To predict the denoising feature C audio 'The boot condition.

[0065] According to one embodiment, the diffusion model includes a series of attention networks. Temporal features and text features can be concatenated to obtain concatenated features. Multiple attention networks are used to sequentially perform attention operations on the concatenated features and the noisy features, thereby optimizing the noisy features (i.e., denoising).

[0066] According to another embodiment, the diffusion model may further include multiple first attention networks and multiple second attention networks, wherein the first attention network optimizes the noisy features (i.e., denoising) by performing attention operations on the temporal features and the noisy features; the second attention network optimizes the noisy features (i.e., denoising) by performing attention operations on the text features and the noisy features. The noisy features are processed by the first attention network and the second attention network multiple times in sequence to denoise the noisy features, thereby obtaining predicted denoised features.

[0067] Optionally, the diffusion model can also include multiple convolutional networks, where the attention network and the convolutional network are alternately connected in series, so that after each attention network performs an attention operation, the convolutional network performs feature extraction on the optimized noisy features. Then, the next attention network can further perform an attention operation on the features processed by the convolutional network.

[0068] During the training process, the original audio feature C can be calculated audio and the predicted denoised features C audio ' similarity between (for example, calculating the audio feature C audio and the predicted unnoised audio features C audio ') to optimize the diffusion model.

[0069] In order to simplify the model training process, Figure 3 The text encoder and audio encoder in

[15] can directly use the currently pre-trained modules. During training, only the parameters of the diffusion model and temporal encoder can be optimized and updated, while the parameters of the text encoder and audio encoder are fixed.

[0070] FIG4 is a schematic flowchart 400 illustrating a method for training a neural network model according to an embodiment of the present disclosure.

[0071] In step S410, a training sample is obtained, where the training sample includes: audio and text description, wherein the text description is used to describe the sound effects contained in the audio and the time when the sound effects are performed.

[0072] According to an embodiment of the present disclosure, sample expansion can be performed on the original sample. Specifically, an original sample can be obtained, wherein the original sample includes: original audio and original text description, wherein the original text description is used to describe the original sound effect contained in the original audio and the time when the original sound effect is performed; then different original samples are combined to obtain a combined sample, and the combined sample is used as the training sample, wherein combining different original samples includes: combining the original audio of the different original samples, and correspondingly combining the original text descriptions of the different original samples.

[0073] For example, the original training sample set may only have samples of "3 seconds of dog barking" and "2 seconds of cat meowing". By combining them, we can get samples of "first 3 seconds of dog barking, then 2 seconds of cat meowing", as well as samples of "1-3 seconds of dog barking, 2-3 seconds of cat meowing", and so on.

[0074] According to an embodiment of the present disclosure, a conjunction can be used to combine the original text descriptions of the different original samples. For example, in the case of connecting the audio of 3 seconds of dog barking and 2 seconds of cat meowing, the conjunction "first... then..." can be used to connect the text descriptions of "3 seconds of dog barking" and "2 seconds of cat meowing" in series to obtain a text description of "first 3 seconds of dog barking, then 2 seconds of cat meowing", and the processed text description corresponds to the audio of first 3 seconds of dog barking, then 2 seconds of cat meowing. For another example, in the case of mixing the audio of 3 seconds of dog barking and 2 seconds of cat meowing, the conjunction "simultaneously" can be used to combine the text descriptions of "3 seconds of dog barking" and "2 seconds of cat meowing" to obtain a text description of "3 seconds of dog barking and 2 seconds of cat meowing at the same time", and the processed text description corresponds to the audio of 3 seconds of dog barking and 2 seconds of cat meowing at the same time (that is, the first 2 seconds of dog barking and cat meowing are performed simultaneously, and the 3rd second is the audio of dog barking).

[0075] It is worth noting that all training samples used in this disclosure comply with the legality, ethics, and privacy requirements of laws and regulations. Specifically, all training samples are sourced from legitimate sources and have been collected with the explicit permission of the user. Furthermore, all training samples used in this disclosure adhere to privacy protection principles, have been rigorously screened and cleaned, and will not be disclosed to any unauthorized third party.

[0076] In step S420, temporal information is extracted based on the text description, wherein the temporal information includes: description information for the sound effect and timestamp information of the sound effect.

[0077] According to an embodiment of the present disclosure, the descriptive information for the sound effect may be the keyword corresponding to the sound effect. For example, if the text description is "2 seconds of bird chirping," the descriptive information for the sound effect may be "bird chirping." The timestamp information for the sound effect may include the start time and the end time of the sound effect.

[0078] According to an embodiment of the present disclosure, the temporal information corresponding to each sample can be made to have a similar format for easy processing. For example, the temporal information corresponding to each sample can sequentially splice the description information for the sound effect and the timestamp information of the sound effect (for example, the start time of the sound effect and the end time of the sound effect). A sample can contain multiple sound effects. For example, in a case where the audio included in the sample is a segment of a dog barking for 3 seconds and then a cat meowing for 2 seconds, the text description corresponding to the audio can be "first 3 seconds of dog barking, then 2 seconds of cat meowing", and the temporal information extracted based on the text description can be "<dog barking><start 0 seconds><end 3 seconds>, ​​<cat meowing><start 3 seconds><end 5 seconds>" (that is, the temporal information corresponding to multiple sound effects is spliced).

[0079] In step S430, the audio, the text description and the temporal information are encoded respectively to obtain audio features, text features and temporal features.

[0080] According to an embodiment of the present disclosure, the audio can be encoded using an audio encoder to obtain the audio features; the text description can be encoded using a text encoder to obtain the text features; and the temporal information can be encoded using a temporal encoder to obtain the temporal features.

[0081] Optionally, during the audio encoding process, Mel feature extraction can be performed on the audio to obtain a Mel spectrum corresponding to the audio; then the Mel spectrum corresponding to the audio can be encoded to obtain the audio features. Mel feature extraction can better simulate the way the human ear processes sound, extract richer audio information, and reduce the dimensionality of the audio features.

[0082] In step S440, the audio feature is denoised to obtain a noisy feature.

[0083] According to an embodiment of the present disclosure, a random noise feature (e.g., random Gaussian noise) may be determined; and a combination ratio of the audio feature and the random noise feature may be randomly determined to generate the noise-added feature. For example, the tth predetermined parameter may be randomly selected from T different predetermined parameters. Where 1≤t≤T and And based on the audio features and The product of the random noise characteristics and The product of determines the noise-added feature (for example, the noise-added feature can be calculated by the above formula (1)).

[0084] In step S450, based on the text feature and the temporal feature, the diffusion model is used to denoise the noisy feature to obtain a denoised feature.

[0085] According to an embodiment of the present disclosure, the diffusion model may include multiple attention networks, and the noisy features may be denoised sequentially via the multiple attention networks.

[0086] For example, Figure 4B As shown, assuming that the number of the multiple attention networks is M, the 1st attention network can be used to optimize (i.e., denoise) the noisy feature to obtain the 1st denoised feature; the m+1th attention network can be used to optimize (i.e., denoise) the mth denoised feature to obtain the m+1th denoised feature; finally, after processing by M attention networks, the processing result of the Mth attention network (i.e., the Mth denoised feature) can be used as the denoised feature, where 1≤m<M.

[0087] According to one embodiment of the present disclosure, in (such as Figure 4B When the diffusion model includes multiple attention networks, two adjacent attention networks in the multiple attention networks can be denoised using the text features and the temporal features respectively. Figure 4C As shown, the first attention network can be used to perform attention operations on the noise-added features and text features to obtain the first denoising feature C 11 Then, the second attention network can be used to denoise the first feature C 11 Perform attention operation on the temporal feature to obtain the second denoising feature C 12 Then the third attention network can be used to denoise the second feature C 12 Perform attention operation on the text feature to obtain the third denoising feature C 13 ...Finally, after being processed by M attention networks, the processing result of the Mth attention network (i.e., the Mth denoising feature) can be used as the denoising feature. It should be understood that although Figure 4C The order of processing text features first and then tense features is used as an example for explanation, but in fact there is no special requirement for the order of processing text features and tense features.

[0088] According to another embodiment of the present disclosure, in (such as Figure 4BWhen the diffusion model includes multiple attention networks, the text features and the temporal features can be spliced ​​to obtain spliced ​​features, and the spliced ​​features can be used to perform denoising on each of the multiple attention networks. Figure 4D As shown, the first attention network can be used to perform attention operations on the noise-added features and the splicing features to obtain the first denoising feature C 21 Then, the second attention network can be used to denoise the first feature C 21 Perform attention operation on the concatenated features to obtain the second denoising feature C 22 Then the third attention network can be used to denoise the second feature C 22 Perform attention operation on the concatenated features to obtain the third denoising feature C 23 ...Finally, after being processed by M attention networks, the processing result of the Mth attention network (i.e., the Mth denoising feature) can be used as the denoising feature.

[0089] According to an embodiment of the present disclosure, the diffusion model may further include multiple convolutional networks, wherein the attention network and the convolutional network are alternately cascaded, and the convolutional network is used to extract features from the processing results of the attention network. That is, after each attention operation is performed by the attention network, the convolutional network performs feature extraction on the processing results of the attention network, and the next attention network performs an attention operation based on the processing results of the attention network.

[0090] In step S460, for each of the samples, the diffusion model is trained based on the difference between the audio feature and the denoising feature.

[0091] According to an embodiment of the present disclosure, a loss function value can be calculated based on the similarity between the audio features and the denoising features; and the diffusion model is trained based on the loss function value. The similarity between the audio features and the denoising features can be calculated in various ways (for example, by calculating the mean square error between the two), which are not limited here.

[0092] According to an embodiment of the present disclosure, a temporal encoder for encoding the temporal information can be trained together with the diffusion model. To simplify the training process, the audio encoder for encoding the audio and the text encoder for encoding the text description can use pre-trained encoders. During the training process, the parameters of the audio encoder for encoding the audio and the parameters of the text encoder for encoding the text description can be fixed.

[0093] Figure 5is a schematic diagram illustrating an audio generation process (ie, an inference process of a neural network model for generating audio) according to an embodiment of the present disclosure.

[0094] like Figure 5 As shown, during the inference process, the neural network model for generating audio can predict the required audio based on random noise features and a text description of the audio to be generated, wherein the text description is used to describe the sound effects contained in the audio to be generated, and the time when the sound effects are performed. The present disclosure obtains the temporal information corresponding to the text description based on the text description, and encodes the text description and the temporal information respectively to obtain text features and temporal features. The text features and temporal features are used as guiding conditions for the input of the diffusion model. The diffusion model can denoise the random noise features to obtain the predicted audio features. Furthermore, by decoding the predicted audio features, the required audio corresponding to the above text description can be obtained.

[0095] Specifically, for the above reasoning process: a text description of the required audio can be input into a computer, and the computer generates a random noise feature as the input of the model. The text description can be either a text description of the audio to be generated (for example, "generate 3 seconds of bird calls") or a simple description of the scene (for example, "generate the sound effect of a walking scene"). In the case where the text description is a simple description of the scene, a large language model (LLM) can be used to generate a text description of the sound effects contained in the audio to be generated and the time when the sound effects are performed based on the simple description of the scene (for example, a simple description of "generate the sound effect of a walking scene" can be used to generate a text description of "generate 3 seconds of human footsteps and 2 seconds of bird calls").

[0096] When the text description is obtained, a large language model can also be used to analyze the text description to obtain the tense information corresponding to the text description (for example, "<footsteps><start 0 seconds><end 3 seconds>, ​​<bird calls><start 3 seconds><end 5 seconds>").

[0097] Then, the temporal information can be encoded by the temporal encoder to obtain the temporal feature C time The text feature C can be obtained by encoding the text description through the text encoder desc Then, by transforming the random noise feature C noise , temporal feature C time , and text features C desc Input diffusion model for noise characteristics C noise After denoising, we can get the denoising feature C audio-1 , and the denoising feature C audio-1The audio features corresponding to the desired audio can be decoded by an audio decoder (e.g., a VAE decoder) to obtain a predicted Mel spectrum. The predicted Mel spectrum can be decoded by a Mel spectrum decoder to obtain the desired audio for use in movies, game dubbing, human-computer interaction, etc.

[0098] It should be noted that the process of converting original audio into Mel spectrum can be achieved with simple calculations, while the process of restoring audio based on Mel spectrum is more complicated and is usually implemented based on a neural network model. The Mel spectrum decoder can directly use the currently pre-trained module.

[0099] Optionally, in order to improve the quality of the predicted audio, the denoising feature C obtained after being processed by the diffusion model can be used. audio-1 Add noise again on the basis of to get the noise feature C mix-1 Then, add the noise feature C mix-1 , temporal feature C time , and text features C desc Input diffusion model to noise feature C mix-1 After denoising, we can get the denoising feature C audio-2 By repeating the above-mentioned denoising and denoising processes, the denoising features can be further optimized. By using the denoising features after multiple denoising and denoising processes as audio features, more accurate required audio features can be obtained.

[0100] Figure 6 is a schematic flow chart 600 illustrating a method for generating audio according to an embodiment of the present disclosure.

[0101] In step S610, an indication for generating audio is obtained, and a text description for the audio to be generated is determined based on the indication, wherein the text description is used to describe the sound effects contained in the audio to be generated and the time when the sound effects are performed.

[0102] According to an embodiment of the present disclosure, the instruction to generate audio may indicate a text description of the audio to be generated, or the instruction to generate audio may indicate a scene description of the scene reflected by the audio. That is, based on the instruction to generate audio, a text description describing the sound effects included in the audio to be generated and the time when the sound effects are performed can be directly obtained, or based on the instruction to generate audio, a scene description of the scene reflected by the audio can be directly obtained, and the scene description does not directly include information about the sound effects included in the audio to be generated and the time when the sound effects are performed.

[0103] Furthermore, if the instruction to generate audio indicates a scene description for the scene reflected by the audio, a scene description for the scene reflected by the audio can be determined based on the instruction; and the text description can be generated based on the scene description. The process of generating the text description based on the scene description can be implemented using a large language model. In this way, the method of the present disclosure can generate rich audio from a simple scene description, without requiring the text used to generate the audio to contain precise descriptions of the sound effects. For example, a scene description can be input into a large language model, and the large language model can be used to generate a specific text description related to the scene description, describing the sound effects included in the audio to be generated and the duration of the sound effects. For example, if the scene description is "Generate the sound effect of a walking scene," the text description can be "Generate 3 seconds of human footsteps and 2 seconds of birdsong," "Generate 5 seconds of human footsteps and 2 seconds of children playing," and so on. It should be understood that the text description generated based on the scene description can have different forms.

[0104] In step S620, temporal information is extracted based on the text description, wherein the temporal information includes: description information for the sound effect and timestamp information of the sound effect.

[0105] According to an embodiment of the present disclosure, the description information for the sound effect may be a keyword corresponding to the sound effect, and the timestamp information of the sound effect may include the start time and the end time of the sound effect. Optionally, the temporal information corresponding to each sample may have a similar format to facilitate processing.

[0106] In step S630, the text description and the temporal information are encoded respectively to obtain text features and temporal features.

[0107] According to an embodiment of the present disclosure, the text description may be encoded using a text encoder to obtain the text features, and the temporal information may be encoded using a temporal encoder to obtain the temporal features. The audio encoder used to encode the audio and the text encoder used to encode the text description may be pre-trained encoders.

[0108] In step S640 , based on the random noise feature, the text feature and the temporal feature, the random noise feature is processed using a diffusion model to obtain an audio feature.

[0109] It should be understood that Figure 6 The diffusion model in can be based on Figure 3 and Figure 4A The model training process shown is obtained by training.

[0110] According to an embodiment of the present disclosure, the diffusion model may include multiple attention networks. The noise features may be processed sequentially through the multiple attention networks to obtain audio features. For example, the number of the multiple attention networks may be M, and the noise features may be denoised using the first attention network to obtain the first denoised feature; the mth denoised feature may be denoised using the m+1th attention network to obtain the m+1th denoised feature; and the Mth denoised feature may be used as the audio feature, where 1≤m<M.

[0111] According to an embodiment of the present disclosure, for two adjacent attention networks among the multiple attention networks, Figure 4C In a similar manner, the text features and the temporal features are used to perform denoising. Alternatively, the text features and the temporal features are spliced ​​to obtain spliced ​​features; and Figure 4D In a similar manner, for each of the multiple attention networks, denoising is performed using the splicing features.

[0112] Optionally, the diffusion model may further include multiple convolutional networks, wherein the attention network and the convolutional network are alternately cascaded, and the convolutional network is used to extract features from the processing results of the attention network.

[0113] After processing the noise features based on the diffusion model comprising multiple attention networks (and convolutional networks), the audio features can be predicted with high accuracy. Furthermore, to further improve the quality of the predicted audio, the random noise features can be subjected to multiple denoising and denoising processes based on the diffusion model to obtain the audio features.

[0114] For example, the random noise feature can be denoised using the diffusion model to obtain a first denoised feature, and the first denoised feature can be denoised to obtain a first noisy feature; the nth noisy feature can be denoised using the diffusion model to obtain an n+1th denoised feature, and the n+1th denoised feature can be denoised to obtain an n+1th noisy feature; and the Nth noisy feature can be denoised using the diffusion model to obtain an N+1th denoised feature, and the N+1th denoised feature can be used as the audio feature, where 1≤n<N.

[0115] It should be understood that in the above-mentioned N times of noise addition, the random noise features added each time may be different. For example, in each noise addition process, the noise feature to be added (e.g., random Gaussian noise) may be determined; and the combination ratio of the nth denoising feature and the noise feature to be added may be randomly determined to generate the noise addition feature. Specifically, the tth predetermined parameter may be randomly selected from T different predetermined parameters. Where 1≤t≤T and And based on the nth denoising feature and The product of the noise feature to be added and The product of t and n determines the n-th noise feature (for example, the n-th noise feature can be calculated in a manner similar to formula (1)). According to an embodiment of the present disclosure, T=N and t=Nn can be made here. As t increases, The value of can be gradually reduced, that is, during the N noise addition processes, the amount of noise added can be gradually reduced.

[0116] In step S650, the audio features are decoded to obtain the audio to be generated.

[0117] According to an embodiment of the present disclosure, the audio feature can be decoded to obtain a mel spectrum corresponding to the audio feature; and the mel spectrum corresponding to the audio feature can be decoded to obtain the audio. The process of decoding the audio feature and decoding the mel spectrum corresponding to the audio feature can be implemented based on a pre-trained decoding module.

[0118] It can be seen that the method disclosed herein extracts temporal information based on the text description corresponding to the audio to be generated without requiring the user to input additional temporal information, and generates audio based on the text description and temporal information together, thereby effectively improving the accuracy of the generated audio, especially the precision in terms of time.

[0119] Figure 7 FIG7 is a schematic diagram 700 showing the composition of an apparatus for training a neural network model according to an embodiment of the present disclosure.

[0120] According to an embodiment of the present disclosure, the apparatus 700 for training a neural network model may include: a sample acquisition module 710 , an information extraction module 720 , an encoding module 730 , a noise addition module 740 , a noise removal module, and a training module 760 .

[0121] The sample acquisition module 710 may be configured to acquire a training sample, wherein the training sample includes an audio and a text description, wherein the text description is used to describe the sound effects contained in the audio and the time when the sound effects occur.

[0122] Optionally, the sample acquisition module 710 can also be configured to: acquire original samples, the original samples including: original audio and original text description, wherein the original text description is used to describe the original sound effects contained in the original audio and the time when the original sound effects are performed; and combine different original samples to obtain combined samples, and use the combined samples as the training samples, wherein combining different original samples includes: combining the original audios of the different original samples, and correspondingly combining the original text descriptions of the different original samples.

[0123] The information extraction module 720 may be configured to extract temporal information based on the text description, wherein the temporal information includes description information for the sound effect and timestamp information of the sound effect.

[0124] According to the embodiments of the present disclosure, the temporal information corresponding to each sample can be made to have a similar format to facilitate processing.

[0125] The encoding module 730 may be configured to encode the audio, the text description, and the temporal information respectively to obtain audio features, text features, and temporal features.

[0126] According to an embodiment of the present disclosure, the text description may be encoded using a text encoder to obtain the text features, and the temporal information may be encoded using a temporal encoder to obtain the temporal features. The audio encoder used to encode the audio and the text encoder used to encode the text description may be pre-trained encoders.

[0127] The noise adding module 740 may be configured to add noise to the audio feature to obtain a noisy feature.

[0128] According to an embodiment of the present disclosure, the noise addition module 740 may determine a random noise feature; and randomly determine a combination ratio of the audio feature and the random noise feature to generate the noise addition feature.

[0129] The denoising module 750 may be configured to: denoise the noisy features using a diffusion model based on the text features and the temporal features to obtain denoised features.

[0130] According to an embodiment of the present disclosure, the diffusion model may include multiple attention networks, and denoising the noisy features using the diffusion model includes: denoising the noisy features sequentially through the multiple attention networks.

[0131] The training module 760 may be configured to: for each of the samples, train the diffusion model based on the difference between the audio feature and the denoising feature.

[0132] According to an embodiment of the present disclosure, the encoding module 730 may encode the temporal information using a temporal encoder to obtain the temporal features. In this case, the training module 760 may train the diffusion model and the temporal encoder together.

[0133] It should be understood that Figure 7 The apparatus 700 for training a neural network model can be implemented as shown in FIG. Figure 4A The various methods 400 for training a neural network model described herein include the sample acquisition module 710, the information extraction module 720, the encoding module 730, the noise addition module 740, the noise removal module, and the training module 760, which can respectively implement the processing of step S410, step S420, step S430, step S440, step S450, and step S460, and are not described in detail herein.

[0134] Figure 8 FIG8 is a schematic diagram 800 showing the composition of an apparatus for generating audio according to an embodiment of the present disclosure.

[0135] According to an embodiment of the present disclosure, the apparatus 800 for generating audio may include: an acquisition module 810 , an information extraction module 820 , an encoding module 830 , a feature processing module 840 , and a decoding module 850 .

[0136] Among them, the acquisition module 810 can be configured to: obtain an indication of generating audio, and determine a text description for the audio to be generated based on the indication, wherein the text description is used to describe the sound effects contained in the audio to be generated, and the time when the sound effects are performed.

[0137] According to an embodiment of the present disclosure, the acquisition module 810 may determine a scene description for the scene reflected by the audio based on the indication; and generate the text description based on the scene description.

[0138] The information extraction module 820 may be configured to extract temporal information based on the text description, wherein the temporal information includes description information for the sound effect and timestamp information of the sound effect.

[0139] According to an embodiment of the present disclosure, the description information for the sound effect may be a keyword corresponding to the sound effect, and the timestamp information of the sound effect may include a start time and an end time of the sound effect.

[0140] The encoding module 830 may be configured to encode the text description and the temporal information respectively to obtain text features and temporal features.

[0141] According to an embodiment of the present disclosure, the text description may be encoded using a text encoder to obtain the text features, and the temporal information may be encoded using a temporal encoder to obtain the temporal features. The audio encoder used to encode the audio and the text encoder used to encode the text description may be pre-trained encoders.

[0142] The feature processing module 840 may be configured to: based on the random noise feature, the text feature and the temporal feature, process the random noise feature using a diffusion model to obtain an audio feature.

[0143] According to an embodiment of the present disclosure, the feature processing module 840 can use the diffusion model to denoise the random noise feature to obtain the first denoised feature, and add noise to the first denoised feature to obtain the first noisy feature; use the diffusion model to denoise the nth noisy feature to obtain the n+1th denoised feature, and add noise to the n+1th denoised feature to obtain the n+1th noisy feature; and use the diffusion model to denoise the Nth noisy feature to obtain the N+1th denoised feature, and use the N+1th denoised feature as the audio feature, where 1≤n<N.

[0144] The decoding module 850 may be configured to decode the audio features to obtain the audio to be generated.

[0145] According to an embodiment of the present disclosure, the decoding module 850 may be configured to: decode the audio feature to obtain a Mel spectrum corresponding to the audio feature; and decode the Mel spectrum corresponding to the audio feature to obtain the audio.

[0146] The generated audio can include a variety of sound effects in chronological order. The generated audio can be used for movie and game dubbing, audio editing, human-computer interaction, etc.

[0147] It should be understood that the apparatus 800 for generating audio can be implemented as Figure 6 The various methods 600 for generating audio are described. The diffusion model can be trained by the above method 400. The acquisition module 810, the information extraction module 820, the encoding module 830, the feature processing module 840 and the decoding module 850 can respectively implement the processing of step S610, step S620, step S630, step S640 and step S650, which will not be repeated here. The device 800 for generating audio can be located at Figure 1 The server 110 shown may also be located at Figure 1 On the terminal 120 shown.

[0148] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or a controller or other computing device, or some combination thereof.

[0149] For example, the method or apparatus according to the embodiment of the present disclosure may also be implemented by Figure 9 The architecture of the computing device 3000 shown in FIG. Figure 9 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the method provided in the present disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 9 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 9 One or more components of a computing device are shown.

[0150] According to another aspect of the present disclosure, a computer-readable storage medium is also provided. Computer-readable instructions are stored on the computer storage medium. When the computer-readable instructions are executed by a processor, the method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer-readable storage medium in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0151] Embodiments of the present disclosure also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a method according to an embodiment of the present disclosure.

[0152] In summary, the embodiments of the present disclosure provide methods, devices, computer program products, and storage media for training a neural network model, as well as methods, devices, computer program products, and storage media for generating audio.

[0153] The method for training a neural network model disclosed herein includes: obtaining a training sample, wherein the training sample includes: audio and a text description, wherein the text description is used to describe the sound effects contained in the audio and the time when the sound effects are performed; extracting temporal information based on the text description, wherein the temporal information includes: description information for the sound effects and timestamp information of the sound effects; encoding the audio, the text description and the temporal information respectively to obtain audio features, text features and temporal features; denoising the audio features to obtain noisy features; denoising the noisy features using a diffusion model based on the text features and the temporal features to obtain denoised features; and for each of the samples, training the diffusion model based on the difference between the audio features and the denoised features.

[0154] The disclosed method extracts temporal information from the textual description corresponding to the audio to be generated, without requiring additional user input of temporal information. The audio is then generated based on both the textual description and the temporal information, effectively improving the accuracy of the generated audio, particularly in terms of temporal detail. Furthermore, the disclosed method can generate rich audio from simple scene descriptions, without requiring the text used to generate the audio to contain precise descriptions of the sound effects.

[0155] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of the code, and the module, program segment, or a part of the code contains at least one executable instruction for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0156] This disclosure uses specific terms to describe the embodiments of the present disclosure. For example, "first / second embodiment," "one embodiment," and / or "some embodiments" refer to a certain feature, structure, or characteristic associated with at least one embodiment of the present disclosure. Therefore, it should be emphasized and noted that "one embodiment," "an embodiment," or "an alternative embodiment" mentioned two or more times in different locations in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of the present disclosure may be appropriately combined.

[0157] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0158] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and should not be interpreted in an idealized or highly formal sense, unless expressly defined as such herein.

[0159] The above is an illustration of the present invention and should not be considered as limiting thereof. Although several exemplary embodiments of the present invention have been described, it will be readily understood by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present invention. Therefore, all such modifications are intended to be included within the scope of the present invention as defined by the claims. It should be understood that the above is an illustration of the present invention and should not be considered as being limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present invention is defined by the claims and their equivalents.

Claims

1. A method for training a neural network model, comprising: Obtaining a training sample, the training sample comprising: audio and a text description, wherein the text description is used to describe the sound effects contained in the audio and the time when the sound effects occur; Extracting temporal information based on the text description, wherein the temporal information includes: description information for the sound effect and timestamp information of the sound effect; Encoding the audio, the text description, and the temporal information respectively to obtain audio features, text features, and temporal features; Noising the audio feature to obtain a noisy feature; Based on the text feature and the temporal feature, denoising the noisy feature using a diffusion model to obtain a denoised feature; and For each of the samples, the diffusion model is trained based on the difference between the audio feature and the denoised feature.

2. The method according to claim 1, wherein Encoding the temporal information includes: Encoding the temporal information using a temporal encoder to obtain the temporal features; Training the diffusion model includes: The diffusion model and the temporal encoder are trained together.

3. The method according to claim 1, wherein Obtaining training samples includes: Obtaining an original sample, the original sample comprising: original audio and an original text description, wherein the original text description is used to describe an original sound effect contained in the original audio and the time when the original sound effect occurs; and Combining different original samples to obtain combined samples, and using the combined samples as the training samples, The combining of different original samples includes: The original audios of the different original samples are combined, and the original text descriptions of the different original samples are correspondingly combined.

4. The method according to claim 1, wherein Adding noise to the audio feature to obtain a noisy feature includes: Determining random noise characteristics; and A combination ratio of the audio feature and the random noise feature is randomly determined to generate the noise-added feature.

5. The method according to claim 4, wherein: Randomly determining a ratio of the audio feature to the random noise feature to generate the noise-added feature includes: Randomly select the tth predetermined parameter from T different predetermined parameters Where 1≤t≤T and as well as Based on the audio features and The product of the random noise characteristics and The product of determines the noise-added feature.

6. The method of claim 1, wherein: The diffusion model includes multiple attention networks, De-noising the noisy features using the diffusion model includes: de-noising the noisy features sequentially through the multiple attention networks.

7. The method according to claim 6, wherein: The number of the multiple attention networks is M, Denoising the noisy features sequentially through the multiple attention networks includes: Denoising the noisy features using a first attention network to obtain a first denoised feature; Denoising the mth denoised feature using the m+1th attention network to obtain the m+1th denoised feature; and The Mth denoising feature is used as the denoising feature, where 1≤m<M.

8. The method of claim 6, wherein: De-noising the noisy features sequentially through the multiple attention networks includes: for two adjacent attention networks among the multiple attention networks, respectively using the text features and the temporal features to perform de-noising.

9. The method of claim 6, further comprising: Splicing the text feature and the temporal feature to obtain a spliced ​​feature; Denoising the noisy features sequentially through the multiple attention networks includes: For each of the multiple attention networks, denoising is performed using the splicing features.

10. The method according to claim 6, wherein: The diffusion model also includes multiple convolutional networks, The attention network and the convolutional network are alternately cascaded, and the convolutional network is used to extract features from the processing results of the attention network.

11. The method of claim 1, wherein: Training the diffusion model based on the difference between the audio feature and the denoising feature includes: Calculating a value of a loss function based on the similarity between the audio feature and the denoising feature; and The diffusion model is trained based on the value of the loss function.

12. The method of claim 1, wherein: Encoding the audio includes: Performing Mel feature extraction on the audio to obtain a Mel spectrum corresponding to the audio; and The mel spectrum corresponding to the audio is encoded to obtain the audio feature.

13. A method for generating audio, comprising: Obtaining an instruction to generate audio, and determining a text description for the audio to be generated based on the instruction, wherein the text description is used to describe the sound effects included in the audio to be generated and the time when the sound effects are performed; Extracting temporal information based on the text description, wherein the temporal information includes: description information for the sound effect and timestamp information of the sound effect; Encoding the text description and the temporal information respectively to obtain text features and temporal features; Based on the random noise feature, the text feature, and the temporal feature, the random noise feature is processed using a diffusion model to obtain an audio feature; and The audio features are decoded to obtain the audio to be generated.

14. The method of claim 13, wherein: The random noise features are processed using a diffusion model to obtain audio features including: Denoising the random noise feature using the diffusion model to obtain a first denoised feature, and adding noise to the first denoised feature to obtain a first noisy feature; Denoising the nth noisy feature using the diffusion model to obtain an n+1th denoised feature, and denoising the n+1th denoised feature to obtain an n+1th noisy feature; and The Nth noisy feature is denoised using the diffusion model to obtain an N+1th denoised feature, and the N+1th denoised feature is used as the audio feature, where 1≤n<N.

15. The method of claim 13, wherein: Determining a text description of the audio to be generated based on the indication includes: determining a scene description for the scene reflected by the audio based on the indication; and The text description is generated based on the scene description.

16. The method of claim 13, wherein: Decoding the audio features includes: Decoding the audio feature to obtain a Mel spectrum corresponding to the audio feature; The mel spectrum corresponding to the audio feature is decoded to obtain the audio.

17. A device for training a neural network model, comprising: The sample acquisition module is configured to: acquire a training sample, wherein the training sample includes: audio and text description, wherein the text description is used to describe the sound effects contained in the audio and the time when the sound effects are performed; An information extraction module is configured to extract temporal information based on the text description, wherein the temporal information includes: description information for the sound effect and timestamp information of the sound effect; an encoding module configured to: encode the audio, the text description, and the temporal information respectively to obtain audio features, text features, and temporal features; a noise adding module, configured to: add noise to the audio feature to obtain a noise adding feature; a denoising module configured to: denoise the noisy features using a diffusion model based on the text features and the temporal features to obtain denoised features; and The training module is configured to: for each of the samples, train the diffusion model based on the difference between the audio feature and the denoising feature.

18. An apparatus for generating audio, comprising: An acquisition module is configured to: acquire an instruction to generate audio, and determine a text description for the audio to be generated based on the instruction, wherein the text description is used to describe the sound effects included in the audio to be generated and the time when the sound effects are performed; An information extraction module is configured to extract temporal information based on the text description, wherein the temporal information includes: description information for the sound effect and timestamp information of the sound effect; An encoding module is configured to: encode the text description and the temporal information respectively to obtain text features and temporal features; a feature processing module configured to: process the random noise feature using a diffusion model based on the random noise feature, the text feature, and the temporal feature to obtain an audio feature; and The decoding module is configured to decode the audio features to obtain the audio to be generated.

19. A computer program product, comprising computer software code, wherein the computer software code is configured to implement the method according to any one of claims 1 to 16 when executed by a processor.

20. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the instructions are used to implement the method according to any one of claims 1 to 16 when executed by a processor.

Images