Audio generation method, device, storage medium, electronic device and program product

Abstract

The embodiments of the present disclosure disclose a method, apparatus, storage medium, electronic device, and program product for audio generation, which encodes the obtained guidance information of the audio generation task to obtain the latent space expression of the guidance information; based on a diffusion model, a hybrid guidance strategy and the latent space expression of the guidance information are used to gradually perform denoising iterations to obtain target data; wherein, the diffusion model includes an original model and a degraded model, and the hybrid guidance strategy is used to guide the original model and the degraded model to perform classifier-free guidance respectively; the target data is decoded into an audio space to obtain the audio information corresponding to the audio generation task. The present disclosure makes the guidance path more structured and robust by using a hybrid guidance strategy that guides the original model and the degraded model to perform classifier-free guidance and degraded model guidance respectively, effectively improving the semantic fidelity and naturalness of the audio signal.

Description

Technical Field

[0001] The present disclosure relates to the fields of computer technology and artificial intelligence technology, and in particular to a method, device, storage medium, electronic device, and program product for audio generation. Background Art

[0002] Audio generation has wide applications in many fields, including but not limited to game development, video production, virtual reality, etc.

[0003] Related technologies often use a classifier-free guidance (CFG) mechanism to guide the diffusion model to generate audio. However, this approach sacrifices sample diversity, resulting in a lack of variability and expressiveness in the generated audio. Summary of the Invention

[0004] Embodiments of the present disclosure provide a method, apparatus, storage medium, electronic device, and program product for generating audio.

[0005] According to one aspect of an embodiment of the present disclosure, there is provided a method for generating audio, comprising:

[0006] Encoding the obtained guidance information of the audio generation task to obtain a latent space representation of the guidance information;

[0007] Based on a diffusion model, a hybrid guidance strategy and a latent space expression of the guidance information are used to gradually perform denoising iterations to obtain target data; wherein the diffusion model includes an original model and a degraded model, and the hybrid guidance strategy is used to guide the original model and the degraded model to perform classifier-free guidance respectively;

[0008] The target data is decoded into an audio space to obtain audio information corresponding to the audio generation task.

[0009] According to another aspect of the embodiments of the present disclosure, there is provided an apparatus for generating audio, including:

[0010] An encoding module, configured to encode the obtained guidance information of the audio generation task to obtain a latent space representation of the guidance information;

[0011] a denoising module, configured to perform denoising iterations step by step based on a diffusion model using a hybrid guidance strategy and a latent space expression of the guidance information to obtain target data; wherein the diffusion model includes an original model and a degraded model, and the hybrid guidance strategy is configured to guide the original model and the degraded model to perform classifier-free guidance, respectively;

[0012] A decoding module is used to decode the target data into an audio space to obtain audio information corresponding to the audio generation task.

[0013] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the above-mentioned audio generation method.

[0014] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device including:

[0015] processor;

[0016] a memory for storing instructions executable by the processor;

[0017] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the above-mentioned audio generation method.

[0018] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer program instructions, wherein when the computer program instructions are executed by a processor, the above-mentioned audio generation method is implemented.

[0019] Based on the above embodiments of the present disclosure, when audio generation is required, the obtained guidance information of the audio generation task can be encoded to obtain the latent space expression of the guidance information; based on the diffusion model, the hybrid guidance strategy and the latent space expression of the guidance information are used to gradually perform denoising iterations to obtain the target data; wherein, the diffusion model includes the original model and the degradation model, and the hybrid guidance strategy is used to guide the original model and the degradation model to perform classifier-free guidance respectively; the target data is decoded into the audio space to obtain the audio information corresponding to the audio generation task. The technical solution disclosed in the present disclosure guides the diffusion model denoising by using the hybrid guidance strategy and guidance information, so that the generated audio signal is closer to the semantic target of the guidance condition, and no additional training is required, thereby greatly improving the stability of audio generation without increasing the complexity of the additional model and the training overhead; in addition, the hybrid guidance strategy of guiding the original model and the degradation model to perform classifier-free guidance and degradation model guidance respectively makes the guidance path more structured and robust, effectively improving the semantic fidelity and the naturalness of the audio signal.

[0020] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other purposes, features, and advantages of the present disclosure will become more apparent through a more detailed description of the embodiments of the present disclosure in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and are not intended to limit the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0022] The present application can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0023] Figure 1 is a system framework diagram to which the present disclosure is applicable;

[0024] Figure 2 is a flowchart of a method for generating audio provided by an exemplary embodiment of the present disclosure;

[0025] Figure 3 is a flowchart of determining target predicted noise in an audio generation process provided by an exemplary embodiment of the present disclosure;

[0026] Figure 4 is a flowchart of determining target predicted noise in an audio generation process provided by another exemplary embodiment of the present disclosure;

[0027] Figure 5 This is a schematic diagram of generating audio based on a hybrid guidance strategy provided by an exemplary embodiment of the present disclosure. Figure 1 ;

[0028] Figure 6 This is a schematic diagram of generating audio based on a hybrid guidance strategy provided by an exemplary embodiment of the present disclosure. Figure 2 ;

[0029] Figure 7 is a structural diagram of an audio generation device provided by an exemplary embodiment of the present disclosure;

[0030] Figure 8 is a structural diagram of an audio generation device provided by another exemplary embodiment of the present disclosure;

[0031] Figure 9 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0033] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0034] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meanings, nor do they indicate a necessary logical order between them.

[0035] It should also be understood that in the embodiments of the present disclosure, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0036] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0037] In addition, the term "and / or" in this disclosure is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the related objects are in an "or" relationship.

[0038] It should also be understood that the description of the various embodiments in this disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced with each other. For the sake of brevity, they will not be described one by one.

[0039] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0040] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0041] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0042] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0043] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate in conjunction with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, among others.

[0044] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system-executable instructions (such as program modules) executed by the computer system. Generally, program modules can include routines, programs, object programs, components, logic, data structures, and the like that perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked via a network. In distributed cloud computing environments, program modules can be located on local or remote computer system storage media, including storage devices.

[0045] Overview of the Disclosure

[0046] Current approaches to audio generation using diffusion models typically utilize a CFG mechanism to guide the generation of audio. However, in these approaches, the CFG mechanism provides guidance based solely on the difference between conditional and unconditional scores. This guidance process lacks fine-grained priors or structural cues, and lacks effective structural feedback or external correction mechanisms. This results in the generated audio being prone to missing details, abnormal reverberation, and unstable rhythms. Furthermore, the CFG mechanism's guidance process relies heavily on the strength of the classifier-free guidance. Improperly setting the strength of the classifier-free guidance can cause inference oscillation or deviation from the textual semantics, hindering improvements in audio generation quality. Furthermore, the CFG mechanism sacrifices sample diversity in the process of compressing the sampling distribution to exclude outliers, which can lead to a lack of variability and expressiveness in the generated audio. Consequently, the CFG mechanism is unable to meet the demands of balancing both quality and diversity in complex audio generation tasks.

[0047] The present disclosure encodes the obtained guidance information of the audio generation task to obtain the latent space expression of the guidance information; then, based on the diffusion model, the hybrid guidance strategy and the latent space expression of the guidance information are used to gradually perform denoising iterations to obtain the target data; wherein, the diffusion model includes the original model and the degradation model, and the hybrid guidance strategy is used to guide the original model and the degradation model to perform classifier-free guidance respectively; the target data is decoded into the audio space to obtain the audio information corresponding to the audio generation task. By hybridizing the classifier-free guidance strategy and the degradation model self-guidance strategy, the present disclosure can not only ensure the semantic alignment of the generated audio, but also ensure that the audio generation process has effective structural feedback, can correct the misleading bias of the diffusion model, and improve the quality and expressiveness of the audio generation.

[0048] Exemplary Systems

[0049] Figure 1 An exemplary system architecture is shown to which the audio generation method or audio generation apparatus according to the embodiments of the present disclosure can be applied.

[0050] like Figure 1 As shown, the system architecture may include a terminal device 101, a network 102, an input device 103, and an audio output device 104. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the audio input device 103. The network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0051] A user can use terminal device 101 to interact with input device 103 and audio output device 104 via network 102 to receive or send audio, etc. Various communication client applications can be installed on terminal device 101, such as artificial intelligence (AI) content generation applications (such as AI videos or AI images), multimedia applications, search applications, web browser applications, shopping applications, instant messaging tools, etc.

[0052] The terminal device 101 can be any electronic device that can deploy a diffusion model for generating audio signals based on a hybrid guidance strategy, including but not limited to mobile terminals such as mobile phones, laptops, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), etc., as well as fixed terminals such as digital televisions, desktop computers, smart home appliances, etc.

[0053] The input device 103 may be a device that provides guidance information. The audio output device 104 is a device that generates high-quality audio signals.

[0054] It should be noted that the audio generation method provided in the embodiments of the present disclosure can be executed by the terminal device 101, and accordingly, the audio generation apparatus can be set in the terminal device 101.

[0055] The audio generation method provided in the embodiment of the present disclosure may be executed by the terminal device 101 or by another server.

[0056] It should be understood that Figure 1 The number of terminal devices 101, network 102, input devices 103, and audio output devices 104 in the above description is merely illustrative. Any number of terminal devices 101, network 102, input devices 103, and audio output devices 104 may be provided as needed. For example, if audio super-resolution processing does not require remote processing, the above system architecture may include only terminal devices 101, excluding the network and server.

[0057] Exemplary Methods

[0058] Figure 2 FIG. 1 is a flow chart of an audio super-resolution processing method provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices such as Figure 1 On mobile devices, such as Figure 2 As shown, the following steps are included:

[0059] Step 201: Encode the obtained guidance information of the audio generation task to obtain a latent space expression of the guidance information.

[0060] Among them, audio generation tasks may include but are not limited to the following tasks: audio generation and repair tasks, onomatopoeic audio generation tasks, and audio enhancement tasks.

[0061] For example, in music restoration tasks, it is necessary to restore or create highly structural audio details, such as harmonics, rhythm, and spatial sense; or, in virtual scenes, such as games or film and television production scenes, it is necessary to create onomatopeia audio such as environmental sounds, action sounds, and emotional sounds; or, for users with hearing impairments or audio equipment optimization scenarios, it is necessary to improve the clarity or customize the style of existing audio.

[0062] The audio generation solution provided by the embodiments of the present disclosure is applicable to various scenarios requiring audio generation and is not specifically limited thereto.

[0063] The guidance information for the audio generation task refers to the information used to control the audio generation process of the diffusion model (DMs) during the inference phase of the diffusion model, thereby obtaining the desired output audio. The diffusion model generates audio waveforms through a step-by-step denoising approach, which has significant advantages in sample quality and stability. The guidance information may include text information, reference images, video information, or other conditional information. Exemplarily, using text information as guidance information means controlling the audio generation process of the diffusion model by providing a text description. For example, by inputting a text description, audio that conforms to the description information is generated; using a reference image as guidance information means inputting one or more images to guide the model to generate audio that conforms to the content or scene described by the reference image; and using a reference video as guidance information means inputting a reference video to guide the model to generate audio that conforms to the content or scene described by the video.

[0064] The latent space expression of the guidance information is a latent space expression obtained by encoding the guidance information using an encoder, such as a variational auto-encoder (VAE) or a Transformer encoder.

[0065] In step 202, based on the diffusion model, a hybrid guidance strategy and a latent space expression of guidance information are used to gradually perform denoising iterations to obtain target data; wherein the diffusion model includes an original model and a degraded model, and the hybrid guidance strategy is used to guide the original model and the degraded model to perform classifier-free guidance respectively.

[0066] Among them, the original model is the fully trained main diffusion model Degrade the model to a poorly trained model , such as under-trained models, low-capacity models. For example, after training, the main diffusion model is obtained In the process of , a degraded model can be obtained by reducing the number of training rounds or reducing the number of parameters. .

[0067] Classifier-free guidance is used to indicate the guidance method that balances the quality and diversity of generated samples in the diffusion model. By adjusting the strength of the classifier-free guidance, a trade-off between sample quality and diversity can be achieved. Because classifier-free guidance does not require the introduction of additional classifiers or supervisory structures, it can be used to perform conditional and unconditional forward inference using the same diffusion model, allowing interpolation and fusion at each sampling step through the strength of the classifier-free guidance.

[0068] In this implementation, the hybrid guidance strategy is a downgraded model guidance that combines classifier-free guidance and downgraded model guidance. The "downgraded model" is used as an auxiliary guidance source to provide additional direction correction in the inference stage. That is, gradient information is constructed through the prediction difference between the downgraded model and the original model, which effectively improves the stability and exploration ability of the diffusion model under extreme input or uncertain conditions, and provides a scalable paradigm for multi-model guidance strategies.

[0069] Degraded model guidance through a degraded model is a method of jointly referencing the output of an insufficiently trained degraded model, such as an undertrained model, a precision-limited model, or a simplified structure model, with a fully trained original model to ensure that the original model identifies and avoids potential low-quality areas based on the output of the degraded model, thereby improving the robustness and detail-preserving ability of the generated audio signal. In the disclosed embodiments, a hybrid guidance strategy is used to combine the results of classifier-free guidance and degraded model guidance during the sampling phase to construct a more robust and detail-preserving sampling trajectory. This effectively alleviates the sample pattern collapse problem caused by the CFG mechanism and solves the loss of diversity caused by the CFG mechanism's overemphasis on conditional scores, improving the model's performance in tasks such as complex text prompts and generalized sound effect generation. Furthermore, the disclosed embodiments perform classifier-free guidance on the original model and the degraded model separately, that is, using different models to generate conditional and unconditional scores simultaneously. This provides an effective structural feedback or external correction mechanism during the sampling phase, thereby correcting misleading biases in the model and effectively improving the quality of audio generation. Moreover, while maintaining the architecture of the existing CFG mechanism unchanged, it achieves "symmetrical correction" of the misguided direction, increases the contrast and dynamic response capability of the guidance path, and strengthens the modeling capabilities of boundary semantics, background environment and sound consistency.

[0070] In the disclosed embodiments, a hybrid guidance strategy and a latent space representation of guidance information are used to generate prediction noise for each sampling step. By iteratively removing the prediction noise corresponding to each sampling step, target data can be generated. The target data is the latent space representation of the generated audio information.

[0071] Among them, the hybrid guidance strategy and the latent space expression of the guidance information are used to determine the prediction noise corresponding to each sampling step, and the denoising iteration process is gradually carried out. Figure 3 and Figure 4 The illustrated embodiment will not be described in detail here.

[0072] Step 203: Decode the target data into an audio space to obtain audio information corresponding to the audio generation task.

[0073] In the disclosed embodiment, a pre-trained VAE decoder may be used to decode the target data to obtain corresponding audio information.

[0074] Based on the embodiments of the present disclosure, when audio generation is required, the obtained guidance information of the audio generation task can be encoded to obtain a latent space expression of the guidance information; based on the diffusion model, the hybrid guidance strategy and the latent space expression of the guidance information are used to gradually perform denoising iterations to obtain the target data; wherein, the diffusion model includes the original model and the degraded model, and the hybrid guidance strategy is used to guide the original model and the degraded model to perform classifier-free guidance respectively; the target data is decoded into the audio space to obtain the audio information corresponding to the audio generation task. The technical solution disclosed in the present invention guides diffusion model denoising by utilizing a hybrid guidance strategy and guidance information, that is, by improving the sampling of the diffusion model through a hybrid guidance method combining degraded model guidance and classifier-free guidance, so that the generated audio signal can not only be close to the semantic target of the guidance condition, but also be compatible with the degraded model guidance to provide effective structural feedback or external correction mechanism, so that the audio generation process is more robust and detail-preserving, effectively alleviating the problems of missing details, abnormal reverberation, and unstable rhythm caused by classifier-free guidance, and avoiding the unexpected interference that may be introduced by the degraded model guidance, resulting in noise, rhythm disorder or spatial illusion in the audio; in addition, since the hybrid guidance strategy is carried out in the inference stage (that is, the sampling stage), no retraining, fine-tuning or additional network parameters are required, which greatly reduces the difficulty of engineering deployment, is suitable for post-insertion enhancement of any diffusion model architecture, and is highly versatile and feasible. Therefore, the embodiments of the present disclosure achieve a significant improvement in the stability of audio generation without increasing the complexity of the model and the training overhead. Moreover, the hybrid guidance strategy of guiding the original model and the degraded model to perform classifier-free guidance respectively makes the guidance path more structured and robust, effectively improving the semantic fidelity and naturalness of the audio signal.

[0075] Figure 3 This is a training flow chart of iterative denoising in the audio generation process provided by an exemplary embodiment of the present disclosure. This embodiment of the disclosure takes how to determine the target predicted noise corresponding to each sampling step by a hierarchical guidance method as an example. Figure 3 As shown, the following steps are included:

[0076] Step 301 : Based on the latent space expression of the original model and the guidance information, a classifier-free guidance is performed to determine the first prediction noise corresponding to each sampling step.

[0077] The first predicted noise corresponding to each sampling step is used to indicate the noise determined without classifier guidance based on the original model.

[0078] In the embodiment of the present disclosure, the first conditional noise corresponding to each sampling step can be predicted based on the latent space expression of the original model and the guidance information; the first unconditional noise corresponding to each sampling step can be predicted based on the original model; and then the first predicted noise can be determined according to the first conditional noise and the first unconditional noise based on the classifier-free guidance strength of the original model.

[0079] For example, each sampling step can be determined by formula (1) t The first prediction noise .

[0080] Formula (1)

[0081] In formula (1), is the first prediction noise, is the sampling step based on the original model prediction t The corresponding first unconditional noise, t is the sampling step, is the sampling step t The corresponding model input vector, c To guide the latent space expression of information, is the sampling step based on the original model prediction t The corresponding first conditional noise, is the classifier-free guidance strength of the original model. The value can be a preset value.

[0082] Step 302 : Based on the latent space expression of the degradation model and the guidance information, a classifier-free guidance is performed to determine the second prediction noise corresponding to each sampling step.

[0083] The second predicted noise corresponding to each sampling step is used to indicate the noise determined without classifier guidance based on the degradation model.

[0084] In the disclosed embodiment, the second conditional noise corresponding to each sampling step can be predicted based on the latent space expression of the degradation model and the guidance information; the second unconditional noise corresponding to each sampling step can be predicted based on the degradation model; and then the second predicted noise can be determined according to the second conditional noise and the second unconditional noise based on the classifier-free guidance strength of the degradation model.

[0085] For example, each sampling step can be determined by formula (2) t The second prediction noise .

[0086] Formula (2)

[0087] In formula (2), is the sampling stept The second prediction noise , is the sampling step based on the degradation model prediction t The corresponding second unconditional noise, t is the sampling step, is the sampling step t The corresponding model input vector, c To guide the latent space expression of information, is the sampling step for prediction based on the degradation model t The corresponding second conditional noise, is the classifier-free guidance strength of the degradation model. The value can be a preset value.

[0088] It is understandable that by adjusting The value of can correspond to different second prediction noises. For example, if is 1, then the second prediction noise corresponds to ,if is 0, then the second prediction noise corresponds to ,if For other values, the second prediction noise is based on the sampling step predicted by the degradation model t The corresponding unconditional noise and conditional noise are obtained.

[0089] Step 303: Obtain a target prediction noise corresponding to each sampling step based on the first prediction noise and the second prediction noise.

[0090] In the embodiment of the present disclosure, after performing classifier-free guidance based on the original model and the degraded model in steps 301 and 302 respectively to obtain the first prediction noise and the second prediction noise corresponding to each sampling step, the prediction noises determined by the classifier-free guidance of the two paths of the original model and the degraded model can be combined to obtain the target prediction noise for each sampling step.

[0091] Specifically, based on the guidance strength of the degraded model in the original model, the target prediction noise corresponding to each sampling step can be obtained according to the first prediction noise and the second prediction noise.

[0092] For example, each sampling step can be determined by formula (3) t Target prediction noise .

[0093] Formula (3)

[0094] In formula (3), For each sampling step t The target prediction noise, t is the sampling step, is the sampling step t The corresponding model input vector, c To guide the latent space expression of information; is the sampling step for unclassified guided prediction based on the degradation model t The corresponding second prediction noise, is the sampling step for non-classified guided prediction based on the original model t The corresponding first prediction noise, is the guidance strength of the degraded model in the original model. The value can be a preset value.

[0095] The above formula (1) example provides an implementation method for determining the first prediction noise, the formula (2) example provides an implementation method for determining the second prediction noise, and the formula (3) example provides an implementation method for determining the target prediction noise based on the first prediction noise and the second prediction noise. Based on the description of the embodiments of the present disclosure, those skilled in the art can know that similar implementation methods can be used to determine the implementation methods of the first prediction noise, the second prediction noise and the target prediction noise for each sampling step, which will not be repeated here.

[0096] In step 304, target data is obtained by performing step-by-step denoising iterations based on the target prediction noise corresponding to each sampling step.

[0097] For the above example in the embodiment of the present disclosure, after each sampling step t The denoising of can be expressed by the following formula (4):

[0098] Formula (4)

[0099] In formula (4), t is the sampling step, is the sampling step t The corresponding model input vector, is the sampling step t The corresponding model output vector, 、 、 It is a parameter used to control noise scheduling and can be obtained through pre-configuration. is a predefined sequence. for , for , represents random noise, is the sampling step t The target prediction noise, represents a standard Gaussian distribution.

[0100] Among them, in the embodiment of the present disclosure, a hybrid guidance strategy and latent space expression of guidance information are used to determine the target prediction noise of each sampling step, and then the target prediction noise corresponding to each sampling step is iteratively removed to generate target data.

[0101] Based on the embodiments of the present disclosure, by first performing classifier-free guidance on the original model and the degraded model respectively to determine the corresponding first prediction noise and second prediction noise, the prediction noises determined by the classifier-free guidance of the two paths of the original model and the degraded model are combined to obtain the target prediction noise of each sampling step. The stability of the generation path is improved through phased branch reinforcement, and the overfitting and pattern collapse problems existing in the existing CFG mechanism are significantly alleviated; by combining hybrid hierarchical guidance with classifier-free guidance and degraded model guidance, multi-guidance direction fusion is achieved, breaking through the reasoning limitations brought by the traditional single guidance direction, and thus being able to greatly improve the audio generation quality while maintaining the freedom of reasoning, and improve the performance of the diffusion model in tasks such as complex text prompts and generalized sound effect generation. At the same time, in the inference stage of generating audio signals, the prediction noise determined by the score results of the unclassified guidance and the degraded model guidance is hierarchically integrated. Under the premise of keeping the unclassified guidance architecture unchanged, structural feedback and correction are achieved through the degraded model guidance, which can systematically solve the diversity loss problem caused by the unclassified guidance and improve the audio effect of the diffusion model in audio generation tasks such as generalized audio generation. In addition, this hybrid guidance method does not require additional model training, fine-tuning and other operations. It is a training-free structural enhancement method, suitable for post-insertion enhancement of any diffusion model architecture, and is highly versatile and feasible.

[0102] Figure 4 This is a training flow chart of iterative denoising in the audio generation process provided by another exemplary embodiment of the present disclosure. This embodiment of the disclosure takes the example of how to determine the target prediction noise corresponding to each sampling step by parallel guidance. Figure 4 As shown, the following steps are included:

[0103] Step 401 : Based on the latent space expression of the original model and the guidance information, a classifier-free guidance is performed to determine the first prediction noise corresponding to each sampling step.

[0104] The method of determining the first predicted noise through step 401 can be found in the description of step 301 in the embodiment shown in Figure 3, and will not be described in detail here.

[0105] Step 402 : Determine the third prediction noise corresponding to each sampling step based on the latent space expression of the original model, the degraded model, and the guidance information.

[0106] In the embodiment of the present disclosure, the third prediction noise is the prediction noise obtained by using the degraded model to guide the original model at each sampling step.

[0107] In specific implementation, the first conditional noise corresponding to each sampling step can be predicted based on the latent space expression of the original model and the guidance information. , and the latent space expression based on the degradation model and guidance information, predicting the second conditional noise corresponding to each sampling step , then based on the first conditional noise and the second conditional noise, the third predicted noise is obtained .

[0108] Step 403 : Based on the model guidance strength of the degradation model, the target prediction noise corresponding to each sampling step is obtained according to the first prediction noise and the third prediction noise.

[0109] In the embodiment of the present disclosure, after performing classifier-free guidance on the original model through steps 401 and 402 to obtain the first prediction noise, and performing model guidance on the original model based on the degraded model to obtain the third prediction noise, the prediction noises obtained by the classifier-free guidance and the model guidance can be combined to obtain the target prediction noise for each sampling step.

[0110] For example, each sampling step can be determined by formula (5) t target prediction noise.

[0111] Formula (5)

[0112] In formula (5), t is the sampling step, is the sampling step t The corresponding model input vector, c To guide the latent space expression of information, The first prediction noise , is the third prediction noise, is the model guidance strength of the degraded model, is the target prediction noise. The value can be a preset value.

[0113] Step 404 : According to the target prediction noise corresponding to each sampling step, target data is obtained through step-by-step denoising iterations.

[0114] Among them, in the embodiment of the present disclosure, a hybrid guidance strategy and latent space expression of guidance information are used to determine the target prediction noise of each sampling step, and then the target prediction noise corresponding to each sampling step is iteratively removed to generate target data.

[0115] Based on the embodiments of the present disclosure, after obtaining the first predicted noise by performing classifier-free guidance on the original model and obtaining the third predicted noise by performing model guidance on the original model based on the degraded model, the predicted noises obtained by the classifier-free guidance and the model guidance are combined, and the predicted noises determined by the scoring results of the classifier-free guidance and the degraded model guidance are integrated in parallel in the inference stage of generating the audio signal to obtain the target predicted noise for each sampling step, thereby increasing the contrast and dynamic response capability of the guidance path. In addition, the hybrid guidance method does not require additional model training, fine-tuning and other operations, and can greatly improve the audio generation quality while maintaining the degree of freedom of reasoning, and can improve the robustness and diversity of audio generation under complex guidance conditions, and can also enhance the modeling capabilities of boundary semantics, background environment and sound effect consistency, thereby greatly improving the audio generation quality.

[0116] Figure 5 This is a schematic diagram of generating audio based on a hybrid guidance strategy provided by an exemplary embodiment of the present disclosure. Figure 1 , which illustrates the process of audio generation based on a hybrid guidance strategy. By inputting the text information "Birds are singing and the breeze is blowing the windows" as guidance information, the guidance information is input into the diffusion model. The diffusion model can iteratively generate target data based on different guidance methods (strategies), such as classifier-free guidance (CFG), parallel guidance (PG) strategy that combines degraded model (bad model) guidance and classifier-free guidance, and hierarchical guidance (HG) strategy that first performs classifier-free guidance on the degraded model (bad model) and the original model (good model) and then combines the two classifier-free guidance paths; finally, the target data generated by the diffusion model is decoded into the audio space through a decoder to obtain an audio signal.

[0117] against Figure 5 The three guidance methods in Figure 6 The stepwise denoising iterative method in the embodiment shown is based on the standard Gaussian distribution. Start by gradually generating target data , each sampling step in the denoising sampling process can be realized by formula (6).

[0118] Formula (6)

[0119] In formula (6), 、 、 It is a parameter used to control noise scheduling and can be obtained through pre-configuration. is a predefined sequence. for , for , represents random noise, is the sampling step t The corresponding model output vector, For the sampling step t The target prediction noise corresponds to the parallel guidance strategy, is the value in formula (5) , corresponding to the hierarchical guidance strategy, is the , corresponding to the no-classifier bootstrap strategy, is the .

[0120] Figure 6 Indicated in 、 、 The parameters in correspond to the parameters in the above formula (1), formula (2), (3), and formula (5), and will not be repeated here. Figure 5 and Figure 6 It illustrates the process of inferring target data based on different guidance strategies and guidance information of the diffusion model.

[0121] Based on the method for generating audio information based on the hybrid guidance strategy disclosed in the above embodiment, text-to-audio generative reasoning experiments and evaluations were conducted on the test set partition of the AudioCaps dataset. The experimental results show that the method for generating audio signals based on the hybrid guidance strategy outperforms the strategy without classifier guidance in semantic and diversity indicators such as the naturalness and fidelity of the generated audio (Frechet Audio Distance, FAD), the semantic similarity between the text condition and the generated audio (Contrastive Language-Audio Pretraining, CLAP), and the Inception score (Inception Score, IS).

[0122] Among them, the AudioCaps dataset is a multimodal dataset designed specifically for audio captioning tasks, containing about 46,000 audio clips and corresponding text pairs.

[0123] In addition, video-to-audio reasoning experiments and evaluations were conducted on the VGGSound test set. The experimental data once again showed that the method based on the hybrid guidance strategy to generate audio signals also performed better than the non-classifier guidance strategy in terms of semantics, alignment and diversity indicators such as FAD, IS, IBS (ImageBind Score, representing the semantic similarity between video conditions and generated audio) and Align Acc (Temporal Align Accuracy, representing the temporal alignment between video conditions and generated audio).

[0124] Among them, the VGGSound dataset is a large-scale video and audio recognition dataset, specially constructed for tasks such as audio event recognition, multimodal learning (such as audio-video alignment, audio generation), etc.

[0125] Exemplary devices

[0126] Figure 7 FIG is a structural diagram of an audio generation device provided by an exemplary embodiment of the present disclosure, which can be applied to electronic devices such as terminal devices, such as Figure 7 As shown, the device includes:

[0127] An encoding module 71 is used to encode the obtained guidance information of the audio generation task to obtain a latent space representation of the guidance information;

[0128] a denoising module 72 for performing denoising iterations step by step to obtain target data based on a diffusion model using a hybrid guidance strategy and a latent space representation of guidance information; wherein the diffusion model includes an original model and a degraded model, and the hybrid guidance strategy is used to guide the original model and the degraded model to perform classifier-free guidance, respectively;

[0129] The decoding module 73 is used to decode the target data into the audio space to obtain audio information corresponding to the audio generation task.

[0130] Figure 8 FIG. 1 is a structural diagram of an audio generation device provided by another exemplary embodiment of the present disclosure. Figure 8 As shown, in Figure 7 Based on the illustrated embodiment, in some embodiments, the denoising module 72 may include:

[0131] A first determination submodule 721 is configured to determine a first prediction noise corresponding to each sampling step without classifier guidance based on the latent space expression of the original model and the guidance information;

[0132] A second determination submodule 722 is configured to perform classifier-free guidance based on the latent space expression of the degradation model and the guidance information to determine a second prediction noise corresponding to each sampling step;

[0133] A third determination submodule 723 is configured to obtain a target prediction noise corresponding to each sampling step based on the first prediction noise and the second prediction noise;

[0134] The first denoising submodule 724 is used to predict the noise according to the target corresponding to each sampling step, and obtain the target data through step-by-step denoising iterations.

[0135] In some embodiments, the first determination submodule 721 is used to predict the first conditional noise corresponding to each sampling step based on the latent space expression of the original model and the guidance information; predict the first unconditional noise corresponding to each sampling step based on the original model; and determine the first predicted noise according to the first conditional noise and the first unconditional noise based on the classifier-free guidance strength of the original model.

[0136] In some embodiments, the second determination submodule 722 is used to predict the second conditional noise corresponding to each sampling step based on the latent space expression of the degradation model and the guidance information; predict the second unconditional noise corresponding to each sampling step based on the degradation model; and determine the second predicted noise according to the second conditional noise and the second unconditional noise based on the classifier-free guidance strength of the degradation model.

[0137] In some embodiments, the third determining submodule 723 is configured to obtain a target prediction noise corresponding to each sampling step based on the guidance strength of the degraded model in the original model, the first prediction noise, and the second prediction noise.

[0138] In some embodiments, the denoising module 72 may include:

[0139] A fourth determination submodule 725 is configured to perform classifier-free guidance based on the latent space expression of the original model and the guidance information to determine a first prediction noise corresponding to each sampling step;

[0140] a fifth determination submodule 726 , configured to determine a third prediction noise corresponding to each sampling step based on the latent space representation of the original model, the degraded model, and the guidance information;

[0141] A sixth determination submodule 727 is configured to obtain a target prediction noise corresponding to each sampling step based on the model guidance strength of the degradation model, the first prediction noise, and the third prediction noise;

[0142] The second denoising submodule 728 is used to predict the noise according to the target corresponding to each sampling step, and obtain the target data through step-by-step denoising iterations.

[0143] It should be noted that the modules in the present device can be decomposed and / or reassembled, and such decompositions and / or reassemblies should be regarded as equivalent solutions of the present device.

[0144] The exemplary embodiment of this device corresponds to the exemplary method described above, and the relevant contents can be referenced and cited to each other. The beneficial technical effects corresponding to the exemplary embodiment of this device can be referred to the corresponding beneficial technical effects of the exemplary method described above, and will not be repeated here.

[0145] Exemplary electronic devices

[0146] Figure 9 A structural diagram of an electronic device provided in an embodiment of the present disclosure includes at least one processor 91 and a memory 92.

[0147] The processor 91 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 9 to perform desired functions.

[0148] The memory 92 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 91 may execute one or more computer program instructions to implement the audio generation methods and / or other desired functions described in the various embodiments of the present disclosure.

[0149] In one example, the electronic device may further include an input device 93 and an output device 94 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0150] The input device 93 may also include, for example, a keyboard, a mouse, a touch screen, a sound pickup device (such as a microphone array), etc.

[0151] The output device 94 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and its connected remote output devices, etc.

[0152] Of course, to simplify, Figure 9 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application scenarios.

[0153] Exemplary systems, computer program products, and computer-readable storage media

[0154] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to perform the steps in the method of audio generation according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0155] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0156] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps in the method of audio generation according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0157] Computer-readable storage media can be any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0158] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0159] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments are sufficient. For system embodiments, since they largely correspond to method embodiments, their description is relatively simple. For relevant parts, references to the description of the method embodiments are sufficient.

[0160] The block diagrams of the devices, apparatuses, and equipment involved in this disclosure are intended only as illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, and equipment may be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and may be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and may be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and may be used interchangeably therewith.

[0161] The methods and apparatus of the present disclosure may be implemented in many ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Therefore, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0162] It should also be noted that in the apparatus, device, and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0163] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0164] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for audio generation, characterized in that: include: Encoding the obtained guidance information of the audio generation task to obtain a latent space representation of the guidance information; Based on a diffusion model, a hybrid guidance strategy and a latent space expression of the guidance information are used to gradually perform denoising iterations to obtain target data; wherein the diffusion model includes an original model and a degraded model, and the hybrid guidance strategy is used to guide the original model and the degraded model to perform classifier-free guidance respectively, the original model is a fully trained main diffusion model, and the degraded model is a model that is insufficiently trained relative to the original model; The target data is decoded into an audio space to obtain audio information corresponding to the audio generation task.

2. The method according to claim 1, characterized in that The diffusion model is based on the hybrid guidance strategy and the latent space expression of the guidance information, and denoising is performed iteratively step by step to obtain the target data, including: Based on the latent space expression of the original model and the guidance information, performing classifier-free guidance to determine the first prediction noise corresponding to each sampling step; Based on the degradation model and the latent space expression of the guidance information, performing classifier-free guidance to determine the second prediction noise corresponding to each sampling step; Obtaining a target prediction noise corresponding to each sampling step based on the first prediction noise and the second prediction noise; According to the target prediction noise corresponding to each sampling step, the target data is obtained through step-by-step denoising iteration.

3. The method according to claim 2, characterized in that The determining of the first prediction noise corresponding to each sampling step based on the latent space expression of the original model and the guidance information includes: Predicting a first conditional noise corresponding to each sampling step based on the latent space expression of the original model and the guidance information; Based on the original model, predict the first unconditional noise corresponding to each sampling step; The first prediction noise is determined according to the first conditional noise and the first unconditional noise based on the classifier-free guidance strength of the original model.

4. The method according to claim 2, characterized in that The performing of classifier-free guidance based on the latent space expression of the degradation model and the guidance information to determine the second prediction noise corresponding to each sampling step includes: Predicting the second conditional noise corresponding to each sampling step based on the degradation model and the latent space expression of the guidance information; Based on the degradation model, predicting the second unconditional noise corresponding to each sampling step; The second prediction noise is determined according to the second conditional noise and the second unconditional noise based on the classifier-free guidance strength of the degradation model.

5. The method according to claim 4, characterized in that The obtaining, based on the first predicted noise and the second predicted noise, the target predicted noise corresponding to each sampling step includes: Based on the guidance strength of the degraded model in the original model, the target prediction noise corresponding to each sampling step is obtained according to the first prediction noise and the second prediction noise.

6. The method according to claim 1, wherein The diffusion model is based on the hybrid guidance strategy and the latent space expression of the guidance information, and denoising is performed iteratively step by step to obtain the target data, including: Based on the latent space expression of the original model and the guidance information, performing classifier-free guidance to determine the first prediction noise corresponding to each sampling step; Determining a third prediction noise corresponding to each sampling step based on the original model, the degraded model, and the latent space expression of the guidance information; Obtaining, based on the model guidance strength of the degradation model and according to the first prediction noise and the third prediction noise, a target prediction noise corresponding to each sampling step; According to the target prediction noise corresponding to each sampling step, the target data is obtained through step-by-step denoising iteration.

7. An audio generation device, characterized in that: include: An encoding module, configured to encode the obtained guidance information of the audio generation task to obtain a latent space representation of the guidance information; a denoising module, configured to perform denoising iterations step by step based on a diffusion model using a hybrid guidance strategy and a latent space representation of the guidance information to obtain target data; wherein the diffusion model includes an original model and a degraded model, the hybrid guidance strategy is configured to guide the original model and the degraded model to perform classifier-free guidance, respectively, the original model being a fully trained main diffusion model, and the degraded model being a model that is insufficiently trained relative to the original model; A decoding module is used to decode the target data into an audio space to obtain audio information corresponding to the audio generation task.

8. A computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 6.

9. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method according to any one of claims 1 to 6.

10. A computer program product comprising computer program instructions, characterized in that When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images