Audio generation method, apparatus, device, storage medium, and program product

Abstract

The present disclosure provides an audio generation method, device, equipment, storage medium and program product, relating to the technical field of computer. The method comprises: determining material music audio and music control information; determining audio noise features according to the material music audio; performing multi-dimensional feature extraction according to the music control information to obtain global guide features and timing guide features; and performing audio reconstruction according to the global guide features, the timing guide features and the audio noise features to generate target audio matching the music control information. The method can make full use of the material music audio, capture and integrate the user's personalized music creation intention based on the music control information, guide through the global guide features and the timing guide features, so that the target audio can not only present the user's desired music expression in a macroscopic way, but also achieve the expected effect in a fine and precise way at a micro local time point, thereby realizing the flexibility, personalization and precision of audio creation.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an audio generation method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the development of computer technology and audio processing technology, audio content plays a crucial role in numerous fields such as multimedia, games, film and television, and education. High-quality, personalized audio not only enriches people's entertainment lives but also becomes a key factor in enhancing user experience and improving the effectiveness of information delivery. Among these technologies, audio generation technology is crucial for personalized creation and diversified audio content. It not only meets users' urgent needs for high-quality, highly expressive audio but also drives innovation and development in various business areas.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this disclosure is to provide an audio generation method, apparatus, device, storage medium, and program product.

[0005] According to a first aspect of the present disclosure, an audio generation method is provided, comprising: determining source music audio and music control information; determining audio noise features based on the source music audio; performing multi-dimensional feature extraction based on the music control information to obtain global guidance features and temporal guidance features; and reconstructing audio based on the global guidance features, the temporal guidance features, and the audio noise features to generate a target audio that matches the music control information.

[0006] In some embodiments, the music control information includes text representing the music type and waveform images representing the audio signal; wherein, multi-dimensional feature extraction based on the music control information to obtain global guidance features and temporal guidance features includes: extracting text-dimensional features based on the text to obtain the global guidance features; extracting features in at least one temporal dimension based on the waveform image to obtain temporal features of the corresponding temporal dimension; and determining the temporal guidance features based on the temporal features of each temporal dimension.

[0007] In some implementations, determining the text includes: determining the text based on first data input by a user for the text dimension; wherein the first data includes at least one of the following: first music audio, music genre description text; the music genre includes at least: music genre, music mood.

[0008] In some implementations, determining the waveform image includes: determining at least one time-series dimension in response to a user's selection operation of a time-series dimension among multiple candidate time-series dimensions; and determining a waveform image for feature extraction of the corresponding time-series dimension based on second data input by the user for the time-series dimension; wherein the second data includes at least one of the following: a second music audio, an audio waveform graph, a curve plotting image, and music time-series description text.

[0009] In some embodiments, the at least one temporal dimension includes a melody dimension, and the waveform image includes a first waveform image; wherein, extracting features of at least one temporal dimension based on the waveform image to obtain temporal features of the corresponding temporal dimension includes: performing frame-by-frame time-frequency conversion on the first waveform image to obtain first frequency domain data of multiple frames; mapping the frequency domain energy in each first frequency domain data to the pitch in a preset pitch set, and determining the pitch features of each frame based on the pitch; performing dimensional conversion on the pitch features of each frame based on a preset number of dimensions to obtain temporal features of the melody dimension.

[0010] In some embodiments, the at least one temporal dimension includes dynamic intensity, and the waveform image includes a second waveform image; wherein, extracting features of at least one temporal dimension based on the waveform image to obtain temporal features of the corresponding temporal dimension includes: performing frame-by-frame time-frequency conversion on the second waveform image to obtain second frequency domain data of multiple frames; performing smooth summation on the frequency domain energy related to each second frequency domain data to obtain a smooth summation result, mapping the smooth summation result to a decibel value; and determining the temporal features of dynamic intensity based on the decibel value corresponding to each frame.

[0011] In some embodiments, the at least one temporal dimension includes a beat dimension, and the waveform image includes a third waveform image; wherein, extracting features of at least one temporal dimension from the waveform image to obtain temporal features of the corresponding temporal dimension includes: performing frame-by-frame time-frequency conversion on the third waveform image to obtain third frequency domain data for multiple frames; performing beat detection on each third frequency domain data to obtain beat detection results, wherein the beat detection results are one of the following: beat points and non-beat points; and determining the temporal features of the beat dimension based on the beat detection results corresponding to each frame.

[0012] In some implementations, determining the timing guidance feature based on the timing features of each timing dimension includes: determining the dimensional order of the at least one timing dimension; sequentially performing incremental compression and superposition on the timing features of each timing dimension according to the dimensional order to obtain at least one superimposed feature; wherein the number of superimposed features is the same as the number of timing dimension types; and fusing the at least one superimposed feature to obtain the timing guidance feature.

[0013] In some implementations, determining the dimensional order of the at least one time-series dimension comprises: determining the dimensional order of the at least one time-series dimension according to a preset dimensional priority; and / or, in response to receiving a sorting operation from a user on the at least one time-series dimension, determining the dimensional order of the at least one time-series dimension.

[0014] In some implementations, audio reconstruction based on the global guidance features, the temporal guidance features, and the audio noise features to generate target audio that matches the music control information includes: inputting the global guidance features, the temporal guidance features, and the audio noise features into a trained audio generation model; wherein the trained audio generation model includes a backbone generation model and a control network, the backbone generation model being constructed based on a preset diffusion model; processing the global guidance features and the temporal guidance features through the control network; processing the audio noise features through the backbone generation model, and adding guidance information to the processing of the backbone generation model using the output of the control network; and generating the target audio based on the reconstructed audio features output by the backbone generation model.

[0015] In some embodiments, the audio generation method further includes: determining a target music style model; the target music style model is obtained by adjusting and training the backbone generation model using music audio corresponding to the target music style; and updating the backbone generation model according to the network parameters in the music style model.

[0016] In some implementations, determining a target music style model includes: identifying the music style model as the target music style model in response to the existence of a music style model associated with a user; or, determining the target music style model in response to a user's selection of a target music style from multiple candidate music styles.

[0017] In some implementations, the temporal guidance feature is output by fusing temporal features of at least one temporal dimension using a trained temporal fusion model, and the temporal features of each temporal dimension are output by a corresponding trained temporal feature extraction model; wherein, the audio generation method further includes: determining sample music audio, sample music description text, sample music waveform image, and tag audio matching the sample music audio, sample music description text, and sample music waveform image; determining sample noise features based on the sample music audio; performing text-dimensional feature extraction based on the sample music description text to obtain sample global guidance features; and performing multiple initial temporal feature extraction models on the sample music waveform image using various temporal features. Feature extraction along the temporal dimension yields sample temporal features across multiple temporal dimensions. An initial temporal fusion model fuses these features to obtain sample temporal guidance features. An initial audio generation model processes the global guidance features, temporal guidance features, and noise features to reconstruct the audio and generate predicted audio. Based on the predicted audio and the labeled audio, loss functions are constructed along the text dimension and the multiple temporal dimensions. These loss functions are then used to train the initial temporal feature extraction model, the initial temporal fusion model, and the initial audio generation model, resulting in a trained temporal fusion model, a trained temporal feature extraction model, and a trained audio generation model.

[0018] In some implementations, determining audio noise features based on the source music audio includes: performing emotion recognition on the source music audio to obtain the emotion features of the source music audio; performing audio encoding on the source music audio to obtain the signal features of the source music audio; and performing fusion encoding based on the emotion features and the signal features to obtain the audio noise features.

[0019] In some implementations, determining the source music audio includes: determining the source music audio based on a third music audio input by a user; or, in response to a user's selection of a music attribute, determining the source music audio from a preset music audio based on the selected music attribute.

[0020] According to a second aspect of the present disclosure, an audio generation apparatus is provided, comprising: a determining unit for determining source music audio and music control information; a noise feature generation unit for determining audio noise features based on the source music audio; a guidance feature generation unit for performing multi-dimensional feature extraction based on the music control information to obtain global guidance features and temporal guidance features; and an audio generation unit for performing audio reconstruction based on the global guidance features, the temporal guidance features, and the audio noise features to generate target audio that matches the music control information.

[0021] According to a third aspect of the present disclosure, an electronic device is provided, characterized in that it includes: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described audio generation method.

[0022] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal is enabled to execute an audio generation method, the method comprising: determining source music audio and music control information; determining audio noise features based on the source music audio; performing multi-dimensional feature extraction based on the music control information to obtain global guidance features and temporal guidance features; and performing audio reconstruction based on the global guidance features, the temporal guidance features, and the audio noise features to generate a target audio that matches the music control information.

[0023] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described audio generation method.

[0024] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0025] This disclosure enables the use of source music audio to precisely capture and incorporate the user's personalized music creation intentions based on music control information. Furthermore, through dual guidance of global and temporal guidance features, the generated audio can not only present the user's desired musical expression on a macro level, but also achieve precise expected effects at micro-level local time points. This enables flexibility, personalization, and precision in audio creation, greatly improving the quality and satisfaction of audio generation.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0028] Figure 1 This is a flowchart illustrating an audio generation method according to some embodiments of the present disclosure.

[0029] Figure 2 This is a flowchart illustrating yet another audio generation method according to some embodiments of the present disclosure.

[0030] Figure 3This is a flowchart illustrating the temporal characteristics of the melody dimension in an audio generation method according to some embodiments of the present disclosure.

[0031] Figure 4 This is a flowchart illustrating the temporal characteristics of dynamic intensity in an audio generation method according to some embodiments of the present disclosure.

[0032] Figure 5 This is a flowchart illustrating a temporal feature of the beat dimension in an audio generation method according to some embodiments of the present disclosure.

[0033] Figure 6 This is a schematic diagram illustrating the determination of timing guidance features in an audio generation method according to some embodiments of the present disclosure.

[0034] Figure 7 This is a flowchart illustrating the generation of target audio in an audio generation method according to some embodiments of the present disclosure.

[0035] Figure 8 This is a schematic diagram of a model processing framework in an audio generation method according to some embodiments of the present disclosure.

[0036] Figure 9 This is a flowchart illustrating the determination of audio noise features in an audio generation method according to some embodiments of the present disclosure.

[0037] Figure 10 This is an interactive process for generating target audio in an audio generation method according to some embodiments of the present disclosure.

[0038] Figure 11 This is a block diagram illustrating an audio generation apparatus according to some embodiments of the present disclosure.

[0039] Figure 12 This is a block diagram illustrating an apparatus for audio generation according to some embodiments of the present disclosure. Detailed Implementation

[0040] Exemplary embodiments of this disclosure will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0041] The embodiments described below, which are examples of some of the embodiments of this disclosure, do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0042] The specific implementation methods of the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0043] Figure 1 This is a flowchart illustrating an audio generation method according to some embodiments of the present disclosure, such as... Figure 1 As shown, the audio generation method can be applied to electronic devices, including but not limited to terminal devices such as smartphones, smart tablets, wearable devices, desktop computers, laptops, and smart speakers. It can also include server-side devices such as local servers and cloud servers. The server-side device can be deployed in a computer cluster consisting of one computer or multiple computers and can include the following steps.

[0044] In step S110, the source music audio and music control information are determined.

[0045] In this embodiment of the disclosure, the source music audio is a basic or prototype audio that can provide basic material for subsequent audio processing. The source music audio can exist in the form of an audio signal.

[0046] Music control information can include specific instructions or parameters to guide audio generation, determining the direction and characteristics of the generated audio. For example, music control information can provide global control over aspects such as style and emotion, or precise local control over aspects such as tempo, pitch, and rhythm of certain sections of music. Music control information can exist in the form of text or images.

[0047] In an exemplary embodiment, if a user requires modifications to an original audio file, the original audio file can be used as source music audio, and music control information can be determined based on the modification request. For example, if the user wants to make the melody of song A higher, song A can be used as source music audio, and the music control information can be determined to carry the information "higher melody". The meaning of "higher melody" can be expressed using text or another audio clip.

[0048] In step S120, audio noise characteristics are determined based on the source music audio.

[0049] In this embodiment, the source music audio can be processed to extract feature data as audio noise features, allowing the source music audio to participate in subsequent processing as audio noise features. These audio noise features can maintain or simulate certain characteristics of the original audio when generating the target audio, ensuring that subsequent audio generation is based on modifications to the original audio (i.e., the source music audio), avoiding excessive modifications that would result in the final effect being unrelated to the original audio.

[0050] In step S130, multi-dimensional feature extraction is performed based on the music control information to obtain global guidance features and temporal guidance features.

[0051] In this embodiment, the global guidance features can be extracted from the text portion of the music control information, and may include emotional tone (such as cheerful, sad, or exciting), music genre (such as classical, rock, or jazz), and the semantic content of the lyrics (such as the emotions expressed). The global guidance features can provide global, time-independent control guidance for subsequent audio generation, influencing the tone, style, or emotional expression of the generated audio, rather than being limited to a specific point in time.

[0052] Timing guidance features can be extracted from the timing portion of music control information. These features can include the music's tempo (e.g., beats per minute, accent positions), tempo (e.g., overall playback speed), and precise melody (e.g., note sequences and pitches), reflecting the music's temporal variation. Timing guidance features enable subsequent audio generation to match the music control information in time, providing time-varying control guidance for subsequent audio generation.

[0053] In this embodiment of the disclosure, global guidance features and temporal guidance features can jointly determine key elements such as style, emotion, rhythm and melody of the generated audio during the audio generation process, so that the generated audio can not only meet the user's music needs in overall performance, but also achieve the expected effect at local time points.

[0054] In step S140, audio reconstruction is performed based on the global guidance features, the temporal guidance features, and the audio noise features to generate target audio that matches the music control information.

[0055] In this embodiment, the extracted features can be combined and an audio generation algorithm can be used to reconstruct the target audio. During the reconstruction process, global guidance features and temporal guidance features can be fully considered to ensure that the generated audio meets the requirements of music control information. At the same time, audio noise features can be combined to ensure that the generated audio is similar to the source music audio in some aspects.

[0056] In an exemplary embodiment, a trained audio generation model can be used to process global guidance features, temporal guidance features, and audio noise features for audio reconstruction. This trained audio generation model can be configured with a trained noise predictor and a control network. The noise predictor can perform noise cancellation based on audio noise features, and the control network can process global guidance features and temporal guidance features, and use the processed data to constrain and guide the noise cancellation process, thereby generating target audio that matches the music control information.

[0057] As can be seen from the above steps, the audio generation method provided in this disclosure can reasonably utilize the source music audio to obtain audio noise features, and extract global guidance features and temporal guidance features from music control information in multiple dimensions. These features are then used as the user's personalized music creation intention to reconstruct the audio based on the audio noise features, resulting in target audio that matches the music control information. Specifically, the audio noise features can constrain the similarity between the target audio and the original audio; the global guidance features and temporal guidance features can jointly influence the generation of the target audio, providing global control guidance and guidance on factors that change over time. Therefore, this solution can fully utilize the source music audio, accurately capture and integrate the user's personalized music creation intention based on music control information, and through the dual guidance of global and temporal guidance features, enable the generated target audio to not only present the user's desired musical expression on a macro level, but also achieve refined expected effects at micro-level local time points. This achieves flexibility, personalization, and precision in audio creation, greatly improving the quality and satisfaction of audio generation.

[0058] In some embodiments of this disclosure, the music control information includes text for characterizing the music type and waveform images for characterizing the audio signal.

[0059] In this embodiment of the disclosure, the music control information can comprehensively and accurately describe the characteristics of the music (i.e., the target audio) that the user expects to generate.

[0060] Text can intuitively express the type of music, such as "classical music," "rock music," and "folk music," providing high-level semantic information for audio generation and helping to determine the overall musical style and emotional tone. Waveform images are a visual representation of audio signals, focusing on showcasing the characteristics of music from the perspective of physical signals. They can contain rich temporal information, reflecting various details of the audio over time, such as changes in amplitude, frequency, and loudness, providing a more microscopic source of information.

[0061] In an exemplary embodiment, the text may be selected by the user from an optional music type, or it may be obtained by performing music type detection on an audio file input by the user; the waveform image may be extracted from the audio file input by the user, or it may be obtained by transforming a curve image drawn by the user.

[0062] In some embodiments of this disclosure, multi-dimensional feature extraction based on the music control information to obtain global guidance features and temporal guidance features may include: extracting textual features based on the text to obtain the global guidance features; extracting features in at least one temporal dimension based on the waveform image to obtain temporal features of the corresponding temporal dimension; and determining the temporal guidance features based on the temporal features of each temporal dimension.

[0063] In this embodiment of the disclosure, the text can be processed by a trained text encoder to encode the text and obtain global guiding features. The text encoder can be, for example, a partial layer structure in a T5 (Text-To-Text Transfer Transformer) text encoder, a Word2Vec (word embedding) model, a BERT (bidirectional encoder) model, etc., and this disclosure does not limit it to this type.

[0064] Global guiding features can include semantic information describing the music, such as genre, emotion, and theme. These features are global and can influence the overall style and emotional direction of the generated audio, controlling the overall musical style during the audio generation process to better match the user's desired music type. For example, features such as elegance, soothing, and rich harmony can be extracted from textual information like "classical music" and used as global guiding features to guide the subsequent audio generation process.

[0065] In this embodiment of the disclosure, waveform images can be processed by various trained temporal feature extraction models to obtain temporal features of different temporal dimensions. Then, the temporal features of various temporal dimensions are fused to obtain the temporal guidance features. Specifically, multiple temporal features of different temporal dimensions can be extracted from a single waveform image, or temporal features of different temporal dimensions can be extracted from different waveform images.

[0066] Waveform images can contain information about how audio signals change over time. Feature extraction from waveform images can yield temporal features such as beat and pitch changes. These features can reflect the specific patterns of music's changes over time. Different temporal dimensions can represent different time series information. For example, features extracted within different time windows can reflect information such as the tempo, note sequence length, and pitch changes in different parts of the music.

[0067] By fusing the temporal features of various temporal dimensions, information from different time points or time periods extracted from waveform images can be integrated to ensure that the target audio generated on the time series can accurately reflect the time information in the music control information, so that the generated target audio matches the music characteristics expected by the user in the time dimension.

[0068] This disclosure provides a method for obtaining global guidance features and temporal guidance features. By combining two different forms of music control information—text and waveform images—it achieves a description of music from both semantic and physical signal dimensions, resulting in a more comprehensive expression of music, enriching the information sources for audio generation, and contributing to the generation of richer, more diverse, and more user-expected audio. Furthermore, this disclosure also possesses a certain degree of versatility and scalability, applicable to the needs of different users. Users can control the generation of target audio through simple text or by providing more detailed waveform images, satisfying the audio generation needs of users at different levels and with varying levels of expertise. It also meets users' control needs across different time dimensions, making this audio generation method more widely applicable.

[0069] Figure 2 This is a flowchart illustrating yet another audio generation method according to some embodiments of the present disclosure, such as... Figure 2 As shown, the audio generation method may include the following steps.

[0070] Step S210: Determine the source music audio and music control information; wherein the music control information includes text used to characterize the music type and waveform images used to characterize the audio signal.

[0071] Step S220: Determine the audio noise characteristics based on the source music audio.

[0072] Step S230: Extract text-dimensional features based on the text to obtain the global guidance features.

[0073] Step S240: Based on the waveform image, perform feature extraction in at least one time-series dimension to obtain the time-series features of the corresponding time-series dimension.

[0074] Step S250: Determine the timing guidance features based on the timing features of each timing dimension.

[0075] Step S260: Perform audio reconstruction based on the global guidance features, the temporal guidance features, and the audio noise features to generate target audio that matches the music control information.

[0076] The specific implementation methods for each of the above steps have been described in detail in the embodiments of the method, and will not be elaborated here.

[0077] In some embodiments of this disclosure, determining the text includes: determining the text based on first data input by a user for the text dimension; wherein the first data includes at least one of the following: first music audio, music genre description text; the music genre includes at least: music genre, music mood.

[0078] In this embodiment of the disclosure, the user can input the first data through an input control corresponding to the text dimension.

[0079] Users can upload or specify a piece of audio (i.e., the first audio file), and then identify the music type of the audio file, using the identification result as the text. For example, a multimodal recognition model based on LLM (Large Language Models) can be used to process the first audio file to obtain its music type.

[0080] Users can also directly enter text describing the music genre, and this directly entered text will be used as the specified text. Examples of music genre descriptions include "rock" and "classical music."

[0081] In an exemplary embodiment, music type may include various classification methods such as music genre (e.g., rock, jazz, classical, etc.), musical mood (e.g., cheerful, sad, passionate, etc.), instrument type, music style (ethnic music, Western music, Eastern music, etc.), and music performance occasion.

[0082] This disclosure allows users to input information in a flexible and diverse manner, which can be specific music audio or simple music genre description text, making it convenient for users with different needs and different levels of expertise to input music control information.

[0083] In some embodiments of this disclosure, determining the waveform image includes: determining at least one time-series dimension in response to a user's selection operation of a time-series dimension among multiple candidate time-series dimensions; and determining a waveform image for feature extraction of the corresponding time-series dimension based on second data input by the user for the time-series dimension; wherein the second data includes at least one of the following: second music audio, audio waveform graph, curve plotting image, and music time-series description text.

[0084] In this embodiment, multiple candidate temporal dimensions can be provided on a preset page for users to select from. These candidate temporal dimensions may include melody, dynamic intensity, beat, emotion, timbre, tempo, etc. Users can select the temporal dimensions they are interested in or emphasize through page operations (such as clicking, swiping, using selection boxes, etc.). This embodiment allows users to selectively choose temporal dimensions according to their music creation needs, providing guidance for subsequent feature extraction and audio generation.

[0085] In this embodiment of the disclosure, the user can input second data through the input control corresponding to the time series dimension. For example, based on the time series dimension selected by the user, prompt information and input controls can be displayed to encourage the user to input the corresponding second data, so that the user can input the second data through the input controls.

[0086] Users can input audio as a second piece of music, from which waveform images can be extracted as a source for extracting features in the corresponding temporal dimension. This waveform image can contain specific temporal information desired by the user in the current temporal dimension. For example, if a user wants to extract beat features from an upbeat rock music audio track and use it as a reference, then the energetic beats in that audio track will be used in subsequent processing.

[0087] Users can input an audio waveform as secondary data. This audio waveform can be obtained by processing the audio using specialized tools. The audio waveform can show the change in the amplitude of the audio signal over time.

[0088] Users can draw custom curves as secondary data, which can represent the user's expectations for certain temporal characteristics. For example, drawing a curve to represent the ideal beat change can help the system understand the user's needs and try to achieve such rhythm changes in subsequent processing.

[0089] Users can input music timing description text as secondary data, that is, describe the characteristics of the audio over time in text form. The music timing description text is then converted into a waveform image for subsequent processing. For example, the music timing description text could be "the tempo speeds up from the first minute and lasts for 30 seconds." Such text information can provide specific requirements regarding the timing dimension, making it easier to convert into a waveform image and then extract timing features.

[0090] Through the embodiments disclosed herein, users can independently select the timing dimension and input second data in various ways, enabling different users to flexibly select and input information according to their own preferences and understanding of music, thereby achieving highly personalized audio creation. This also allows the solution to adapt to the operating habits and ability levels of different users.

[0091] In some embodiments of this disclosure, the at least one temporal dimension includes a melody dimension, and the waveform image includes a first waveform image.

[0092] Figure 3 This is a flowchart illustrating the temporal characteristics of the melody dimension in an audio generation method according to some embodiments of this disclosure. For example... Figure 3 As shown, in some embodiments of this disclosure, the extraction of melodic dimension features based on the first waveform image to obtain temporal features of the melodic dimension may include the following steps.

[0093] Step S310: Perform frame-by-frame time-frequency conversion on the first waveform image to obtain first frequency domain data of multiple frames.

[0094] In this embodiment, the entire waveform can be divided into multiple shorter frames, each representing a short time segment of the waveform. Then, a time-frequency transformation (such as a Fourier transform) can be performed on each frame to convert the time-domain signal into frequency-domain data. This embodiment can obtain the first frequency-domain data of multiple frames, each frame containing the energy distribution of the signal in the frequency domain within that time period.

[0095] Step S320: Map the frequency domain energy in each first frequency domain data to the pitch in the preset pitch set, and determine the pitch characteristics of each frame based on the pitch.

[0096] In this embodiment of the disclosure, the energy in the first frequency domain data of each frame can be mapped to a preset pitch set. This preset pitch set can be determined based on standard pitches in music theory (such as semitone steps) and represents possible pitch values, for example, it can include 24 pitches corresponding to two octaves starting from middle C.

[0097] In this embodiment, mapping frequency domain energy to pitch transforms complex frequency domain information into more musical pitch information. During the mapping process, the dominant or most prominent pitch corresponding to each frame can be determined based on the distribution of frequency domain energy. That is, the pitch corresponding to the most significant frequency domain energy (i.e., peak value) in each frame can be determined, which can be achieved by finding the maximum value or a value exceeding a certain threshold in the frequency domain data. For example, if the frequency domain energy of a frame is concentrated in a certain frequency range, mapping can be performed using a scale feature function (such as the Chroma function in Librosa) to determine which pitch (e.g., middle C or high G) corresponds to that frequency range. In this way, each frame can be assigned a pitch feature to reflect the pitch information of the signal within that frame.

[0098] Step S330: Perform dimensional transformation on the pitch features of each frame based on a preset number of dimensions to obtain the temporal features of the melody dimension.

[0099] In this embodiment, the preset dimension number can be the feature dimension number required for subsequent audio reconstruction. The dimension number of the pitch feature can be related to the number of elements in the pitch set, and the number of elements can be a different value from the preset dimension number. The pitch features can be converted into temporal features of the melody dimension that match the preset dimension number through a linear mapping, so that the temporal features of the melody dimension can be more easily used for subsequent processing.

[0100] In this embodiment of the disclosure, the temporal features of the melody dimension can be a feature representation of the entire melody in a time sequence. It can integrate the pitch information of different time frames as part of the temporal guidance features, affecting the melody direction and changes of the final generated audio.

[0101] Through the embodiments of this disclosure, temporal features related to melody can be accurately extracted from waveform images, complex audio signals can be converted into quantifiable pitch features, providing accurate time-varying melody guidance information for subsequent audio generation, which helps to accurately control the melody of the target audio.

[0102] In an exemplary embodiment, the melody dimension can be represented using Cmel, and the temporal feature of the melody dimension can be a feature of the form T×64×1; where T represents the frame rate, 64 represents the preset number of dimensions, and 1 represents the number of audio tracks. For the melody dimension, feature extraction can be performed in the following manner:

[0103] Based on the first waveform image, a linear spectrogram (i.e., the first frequency domain data of multiple frames) is determined. The linear spectrogram indicates the range of frame signal energy within B discrete frequency ranges. The B frequency energies can be rearranged frame-by-frame into 24 pitch classes (or semitones, i.e., C, C#, ..., B) using the Chroma function in the Libosa tool. That is, pitch mapping is performed independently for each frame t∈{1,...,T}, resulting in pitch features in a (Tx24x1) dimension.

[0104] Furthermore, the chroma map (i.e., linear spectrogram) can be made to have similar regional distributions through mean calculation, retaining only the most prominent pitch class. In addition, a band-pass filter with a cutoff frequency of middle C, i.e., 240Hz, can be applied before chroma map calculation to prevent bass from dominating.

[0105] Next, the chroma map encoding (i.e., pitch features) generated from the chroma map can be mapped to 64-dimensional Mel features (i.e., temporal features of the melody dimension) using a feature sampler, that is, transforming 24 dimensions into 64 dimensions. This can be achieved through linear processing using a neural network to obtain Mel features in the form of (Tx64x1) dimensions.

[0106] In some embodiments of this disclosure, the at least one time dimension includes dynamic intensity, and the waveform image includes a second waveform image.

[0107] In this embodiment of the disclosure, dynamic intensity is a temporal dimension that describes how signal intensity (or amplitude) changes over time. The second waveform image may be the same as or different from the aforementioned first waveform image.

[0108] Figure 4 This is a flowchart illustrating the temporal characteristics of dynamic intensity in an audio generation method according to some embodiments of this disclosure. For example... Figure 4 As shown, in some embodiments of this disclosure, the dynamic intensity feature extraction based on the first waveform image to obtain the temporal features of the dynamic intensity may include the following steps.

[0109] Step S410: Perform frame-by-frame time-frequency conversion on the second waveform image to obtain second frequency domain data of multiple frames.

[0110] Step S420: Smoothly sum the frequency domain energy related to each second frequency domain data to obtain a smoothed summation result, and map the smoothed summation result to a decibel value.

[0111] In this embodiment of the disclosure, for the second frequency domain data of multiple frames, the energy of these frequency domains can be smoothed and summed. The energy of each frequency component in the frequency domain is added in a smooth manner to obtain a single numerical result as the smoothed summation result. A smoothing filter with a preset duration context window (e.g., a 1-second context window) can be used for energy summation to avoid abrupt changes or discontinuities during energy summation, ensuring that the result better matches the physical characteristics and auditory experience of the audio signal in reality.

[0112] The smoothed summation of frequency domain energy can be mapped to decibels (dB), a unit of measurement for audio intensity that characterizes the loudness of a note. By converting the smoothed summation to decibels, the magnitude of the energy is transformed into a unit of measurement more consistent with human hearing and audio engineering. This step makes the subsequent features more meaningful, as decibel values ​​better reflect the human ear's perception of sound intensity.

[0113] Step S430: Determine the temporal characteristics of dynamic intensity based on the decibel values ​​corresponding to each frame.

[0114] In this embodiment of the disclosure, the decibel value corresponding to each frame can be used as the value in the temporal feature of dynamic intensity. The temporal feature of dynamic intensity can reflect the intensity change of audio over time, i.e., the change in volume, and can also comprehensively reflect the density of notes, providing important temporal information for subsequent audio processing or generation.

[0115] For example, in music, certain parts may crescendo (volume gradually increases) or diminuendo (volume gradually decreases). The temporal characteristics of these decibel values ​​can accurately represent this dynamic intensity change, and thus, in audio processing, the volume changes of the audio can be generated or adjusted based on this information.

[0116] Through the embodiments of this disclosure, the dynamic intensity temporal features of audio can be accurately extracted from waveform images, providing accurate guidance information on volume changes and note density for subsequent audio generation, which helps to control the volume and density of notes in the target audio.

[0117] In an exemplary embodiment, dynamic intensity can be represented using Cdyn, and the temporal features of dynamic intensity can be features of T×1×1 dimension; features of dynamic intensity can be extracted in the following manner:

[0118] Based on the second waveform image, a linear spectrogram (i.e., second frequency domain data from multiple frames) is determined. The frequency energy of each time frame of the linear spectrogram is summed, and the resulting value is mapped to a decibel (dB) scale to obtain the temporal characteristics of dynamic intensity. Dynamic intensity is closely related to human-perceived loudness. To mitigate the rapid fluctuations in the original dynamic value caused by the onset of notes or percussion, and to make dynamic control closer to the perceived music intensity, a smoothing filter with a one-second context window can be applied to each frame to sum the frequency domain energy.

[0119] The temporal characteristics of dynamic intensity are also closely related to some musical attributes (such as instruments, harmonic textures, rhythmic density, etc.). For example, string instruments have smoother dynamic changes when played, while percussion instruments may have more abrupt dynamic changes. Therefore, for the generated target audio, the temporal characteristics of dynamic intensity can not only provide constraints on loudness, but also on the aforementioned musical attributes.

[0120] In some embodiments of this disclosure, the at least one time dimension includes a beat dimension, and the waveform image includes a third waveform image.

[0121] Figure 5 This is a flowchart illustrating the temporal features of the beat dimension in an audio generation method according to some embodiments of this disclosure. Figure 5 As shown, in some embodiments of this disclosure, extracting features in the beat dimension based on the first waveform image to obtain temporal features in the beat dimension may include the following steps.

[0122] Step S510: Perform frame-by-frame time-frequency conversion on the third waveform image to obtain third frequency domain data for multiple frames.

[0123] Step S520: Perform beat detection on each third frequency domain data to obtain beat detection results. The beat detection results are one of the following: beat points and non-beat points.

[0124] In this embodiment, a trained beat detector can process the third frequency domain data of each frame, performing beat detection on each third frequency domain data to identify regular accents or rhythmic points in the music. The beat detector can be implemented based on an LSTM (Long Short-Term Memory) network, which can learn the time dependence of audio signals to accurately detect the beats of the music. The output of the beat detection can be a beat point or a non-beat point, where the beat point is a rhythmic part of the music, corresponding to an accented or emphasized note.

[0125] Beat detection points can also include ordinary beats and accented beats. Ordinary beats are rhythmic points that are relatively evenly distributed and of relatively consistent intensity in music, forming the basic rhythmic framework of the music; accented beats are rhythmic points that are more prominent and of greater intensity in music, playing a more crucial role in the musical structure, such as the strong beats or climaxes of a piece of music.

[0126] Step S530: Determine the temporal features of the beat dimension based on the beat detection results corresponding to each frame.

[0127] In this embodiment, the temporal features of the beat dimension can characterize the position of beat points in the audio, the time interval between beats, the stability of the beats, etc., and can reflect the beat distribution of the audio over time. By recording beat points and non-beat points, a time series representing the musical beat can be obtained, which can be used in audio reconstruction to generate music with an ideal rhythm.

[0128] For example, a third-waveform image can correspond to a 4 / 4 time signature of music. The beat points of this music will appear at certain intervals, and this information will be recorded in the temporal features of the beat dimension, providing a basis for subsequent audio reconstruction so that the generated target audio has the same beat and rhythm as this music.

[0129] Through the embodiments of this disclosure, waveform images can be framed for time-frequency conversion and beat detection, accurately extracting the beat information of the audio, providing precise rhythmic beat guidance information for subsequent audio generation, which helps to accurately control the beat of the target audio, add percussion instruments, change the intensity of the rhythm, or change the beat mode, thereby enhancing the rhythm of the music.

[0130] In an exemplary embodiment, the beat dimension can be represented using Crhy, and the temporal feature of the beat dimension can be a feature of T×2×1 dimension; the beat dimension feature can be extracted in the following way:

[0131] Based on the third-waveform image, a linear spectrogram (i.e., third-frequency domain data from multiple frames) is determined. An internally implemented LSTM-based beat detector, trained on different internal datasets, can be used to predict whether a frame is located at a beat point (which may include regular beats, accented beats), or a non-beat point. Then, frame-by-frame beat and accent probabilities are used for control, resulting in two categories per frame. This time-varying beat / accent control allows creators to precisely synchronize beats / accents, for example, in video scene editing or when there are moments in video content that require syncing with the target audio. Furthermore, this method can encode subtle rhythmic information, such as whether the music sounds more harmonious or rhythmic, whether the rhythmic pattern is clear / simple or complex. Experienced music creators can use this method to finely control the rhythm of the target audio. During the training of the feature extraction model corresponding to the beat dimension, the beat / accent probability curves extracted from sample music can be time-stretched to match the desired rhythm, creating labeled audio for model training. In addition, users can obtain accurate beat / repeat timestamps by inputting the beat / repeat curve into a post-filter based on a Hidden Markov Model (HMM), and use the timestamps to move the curve along the time axis to achieve synchronization.

[0132] In some embodiments of this disclosure, determining the timing guidance feature based on the timing features of each timing dimension includes: determining the dimensional order of the at least one timing dimension; sequentially performing incremental compression and superposition on the timing features of each timing dimension according to the dimensional order to obtain at least one superimposed feature; wherein the number of superimposed features is the same as the number of timing dimension types; and fusing the at least one superimposed feature to obtain the timing guidance feature.

[0133] In this embodiment, the dimension order can be determined in different ways, such as by a preset order, by the order of user operations, or by importance. Clearly defining the dimension order ensures the consistency and orderliness of subsequent processing.

[0134] In this embodiment, the incremental compression and overlay process can involve processing the temporal features of each dimension sequentially, and then overlaying or fusing the processing result of the previous dimension with the temporal features of the current dimension to generate a new overlay feature. In this way, the earlier the temporal dimension in the dimensional sequence, the more guiding it can be in the audio reconstruction.

[0135] For example, suppose there are temporal features A0 in the beat dimension and B0 in the dynamic intensity dimension, with A0 first and B0 last. First, A0 can be compressed to obtain a superimposed feature A1. Then, A1 and B0 are superimposed to obtain a superimposed feature B1 corresponding to B0. Finally, A1 and B1 are fused to obtain the temporal guidance feature.

[0136] In an exemplary embodiment, temporal features from various temporal dimensions can be concatenated using a compression and stacking method, and then fused using two layers of two-dimensional convolution to obtain multiple stacked features. All stacked features are then modeled using a multi-dimensional spatial module, which may include self-attention and convolutional blocks. This module allows for preliminary modeling and fusion of temporal features from different temporal dimensions, enabling the mutual influence of temporal features. After fusion, feature dimension compression can be performed to obtain temporal guided features.

[0137] Through the embodiments of this disclosure, information from multiple dimensions can be effectively integrated into a single temporal guiding feature in dimensional order, avoiding the limitations of single-dimensional information and providing more comprehensive time-series information for subsequent audio processing, thus contributing to the generation of richer and more complete audio. Furthermore, the processing order of different temporal dimensions can be flexibly adjusted according to different needs, providing users with greater flexibility. Users can adjust the focus of audio reconstruction on different music styles and application scenarios, making the generated audio more suitable for specific requirements.

[0138] Figure 6 This is a schematic diagram illustrating the determination of timing guidance features in an audio generation method according to some embodiments of this disclosure. Figure 6 As shown, it includes a first time series feature 601, a second time series feature 602, and a third time series feature 603, whose corresponding time series dimensions are arranged sequentially in the dimension order.

[0139] Specifically, the first timing feature 601 can be input to the first compression module, the second timing feature 602 can be input to the second compression module, and the third timing feature 603 can be input to the third compression module.

[0140] The first superimposed feature output by the first compression module after processing the first timing feature 601 can be obtained; the output of the first compression module is also input to the second compression module to obtain the second superimposed feature output by the second compression module after processing the second timing feature 602 and the first superimposed feature; the output of the second compression module is also input to the third compression module to obtain the third superimposed feature output by the third compression module after processing the third timing feature 603 and the second superimposed feature.

[0141] Finally, the first, second, and third superimposed features are fused to obtain the temporal guidance features.

[0142] In some embodiments of this disclosure, determining the dimensional order of the at least one time-series dimension comprises: determining the dimensional order of the at least one time-series dimension according to a preset dimensional priority; and / or, in response to receiving a sorting operation from a user on the at least one time-series dimension, determining the dimensional order of the at least one time-series dimension.

[0143] In this embodiment, dimensional priorities can be pre-defined. Dimensional priority is a fixed rule or order used to determine the processing order of at least one temporal dimension. Higher-priority temporal dimensions can be placed earlier in the dimensional order. These pre-defined priorities can be based on music theory, rules of thumb, or common audio processing requirements. Such a pre-defined order ensures that, even without explicit user specification, various temporal features are processed in a relatively reasonable order, guaranteeing that the generated audio meets general standards in terms of basic musicality.

[0144] In this embodiment of the disclosure, users can also perform active sorting operations based on different music creation goals. Users can sort the temporal dimensions according to their music creation intentions or specific needs for audio processing to meet the need to process certain temporal dimensions first.

[0145] For example, if a user is creating music characterized by rhythmic variations, the dynamic intensity dimension can be processed first, followed by the beat dimension, and finally the melody dimension. The user can specify the order of these dimensions by sending sorting commands through the interface or configuration files. This gives users more control and flexibility, allowing audio processing to better suit their individual needs.

[0146] In an exemplary embodiment, the dimension order can also be determined simultaneously based on a preset dimension priority and the user's sorting operation. For example, assuming that the at least one time-series dimension includes A, B, and C, and the user can specify only A as the most important time-series dimension, then in the dimension order, time-series dimension A can be ranked first, and then the second and third positions in the dimension order can be determined according to the dimension priorities of time-series dimensions B and C.

[0147] Figure 7 This is a flowchart illustrating the generation of target audio in an audio generation method according to some embodiments of this disclosure. For example... Figure 7 As shown, in some embodiments of this disclosure, audio reconstruction based on the global guidance features, the temporal guidance features, and the audio noise features to generate target audio that matches the music control information may include the following steps.

[0148] Step S710: Input the global guidance features, the temporal guidance features, and the audio noise features into the trained audio generation model; wherein, the trained audio generation model includes a backbone generation model and a control network, and the backbone generation model is constructed based on a preset diffusion model.

[0149] Among them, the diffusion model is a generative model, such as the SD1.5 model (Stable Diffusion v1.5), which generates new data through a process of progressively adding noise and denoising. In the audio domain, the backbone generative model can utilize the principle of the diffusion model, combining the guiding information of the control network when processing audio noise features to generate audio reconstruction features that meet the expectations.

[0150] ControlNet can be used to process global and temporal guidance features, which can contain high-level information about audio content, style, rhythm, etc.

[0151] Step S720: The global guidance features and the timing guidance features are processed through the control network.

[0152] In this embodiment of the disclosure, the control network can process global guidance features and timing guidance features to generate some control signals or guidance information.

[0153] Step S730: The audio noise features are processed by the backbone generation model, and the output of the control network is used to add guiding information to the processing of the backbone generation model.

[0154] In this embodiment of the disclosure, the output of the control network (i.e., guidance information) can be used to guide the processing of the backbone generation model. In this process, the output of the control network can be combined with the intermediate or output layers of the backbone generation model to influence the generated audio features.

[0155] In an exemplary embodiment, the global guidance features can be processed by the control network, and the processed global guidance features and the audio noise features are input into the encoding unit of the backbone generation model for interpolation and concatenation encoding to obtain encoded features. Then, the encoded features and the temporal guidance features are synthesized to obtain synthesized guidance features.

[0156] Step S740: Generate the target audio based on the reconstructed audio features output by the backbone generation model.

[0157] In this embodiment of the disclosure, the synthesis guidance features can be layered and decoded by the decoding unit of the backbone generation model to obtain reconstructed audio features, and then the target audio can be generated based on the reconstructed audio features.

[0158] The reconstructed audio features can be reconstructed spectral data features, which can be decoded using a latent space decoder to obtain reconstructed spectral data. Then, the reconstructed spectral data is subjected to an inverse short-time Fourier transform to obtain the target audio.

[0159] In this embodiment, textual guidance (i.e., global guidance features) can be introduced into the encoding part of the diffusion model Unet noise generator (i.e., the backbone generation model), and temporal guidance features related to timing control can be introduced before the decoding part to participate in guidance. Specifically, the encoder output (i.e., encoded features) can be combined with the temporal guidance features after adding nonlinearity via a sigmoid function. The combination method is X = Norm(X + X·X'). Finally, this synthesized result (i.e., synthesized guidance features) is injected layer by layer into the decoder module to achieve joint guidance control of global guidance features and temporal guidance features.

[0160] Through the embodiments of this disclosure, global guidance features, temporal guidance features, and audio noise features can be combined, and a backbone generation model and a control network can work together to process different types of features separately. Specifically, a backbone generation model based on a diffusion model can be used, leveraging the powerful data generation capabilities of the diffusion model and combining it with the guidance of the control network to control audio generation from multiple dimensions. This improves the richness and accuracy of the generated audio, ensuring that the generated audio has both overall stylistic consistency and excellent temporal performance, while retaining certain original audio characteristics, thus generating target audio with better results.

[0161] In some embodiments of this disclosure, the audio generation method further includes: determining a target music style model; the target music style model is obtained by adjusting and training the backbone generation model using music audio corresponding to the target music style; and updating the backbone generation model according to the network parameters in the music style model.

[0162] In this embodiment of the disclosure, music audios matching different music styles (e.g., classical, rock, jazz, electronic, etc.) can be used as training data in advance to self-train the backbone generation model, thereby obtaining music style models corresponding to different music styles. The target music style model can be one of multiple music style models.

[0163] During training, the backbone generation model can adjust its network parameters based on the features in these music audios, enabling the backbone generation model to learn the unique features of the music style, such as the harmonic structure, melodic direction, and rhythmic characteristics of classical music.

[0164] Once training is complete, the target music style model will contain the trained network parameters. These parameters reflect the characteristics of the target music style. By updating these parameters into the backbone generation model, the backbone generation model can generate audio with the characteristics of the target music style. The updated backbone generation model will then be able to generate audio with the characteristics of the target music style. For example, when generating classical music, it can generate audio according to the stylistic features of classical music, including using appropriate instrument timbres, chord progressions, and rhythmic patterns.

[0165] Through the embodiments of this disclosure, music audio corresponding to the target music style can be used for training, making the generated audio more accurately conform to the characteristics of that music style. This helps to make the generated audio more professional and pure in style, improving the audio quality and musicality. Furthermore, by training music style models for multiple different music styles, the backbone generation model can flexibly possess music style processing capabilities, enabling it to adapt to various music style generation tasks.

[0166] In an exemplary embodiment, for the training portion of the style model, the modeling of the music style can be achieved using published songs with high professional quality; for example, the style model can be fine-tuned and trained using a single music audio file. Alternatively, user-generated audio with lower professional quality, such as humming, a cappella, rhythmic singing, or rap, can be used to achieve personalized style modeling.

[0167] In the early stages of style model training, a portion (e.g., 10%) of the training data from the backbone model can be extracted according to style category to initialize the pre-trained style model. This can be achieved using LoRA (Low-Rank Adaptation) training, with the LoRA model size compressed to a minimum for lightweight ensemble integration and more concise, fast, and effective fine-tuning. Furthermore, for user-recorded audio data, preprocessing operations such as silence detection, audio style detection, and audio frequency domain transformation can be performed before using the processed frequency domain features to fine-tune the pre-trained style model. User information can also be used for ID encoding. This allows for the creation of personal information tags for different users, hiding direct user information; furthermore, the encoded user IDs can be sent to the user's device, and combined with model quantization, this protects user information, ensuring personalized preservation of model weights and reducing the risk of data leakage.

[0168] The training of style models is relatively quick and can be done in the cloud. Style models can be deployed and combined with the backbone model on the device, or they can be used directly in the cloud, with only the model parameters deployed for combination with the backbone model.

[0169] In some embodiments of this disclosure, determining a target music style model includes: determining the music style model as the target music style model in response to the existence of a music style model associated with a user; or, determining the target music style model in response to a user's selection operation of a target music style among multiple candidate music styles.

[0170] In this embodiment, it can be checked whether a music style model associated with the user exists. This model can be one that the user has previously used, created, saved, or that the system has recommended. If such a model exists, it can be directly identified as the target music style model. This embodiment simplifies the user's selection process and quickly adapts to the user's personalized needs.

[0171] In this embodiment, multiple candidate music styles can be provided for the user to choose from. These candidate styles can cover different music genres, such as music styles, periods, artists, or emotional characteristics. Users can select the desired target music style through interactive operations on the interface (such as clicking, dragging, or voice commands). After the user makes a selection, the target music style model can be determined based on the selected style. This embodiment can provide greater flexibility, allowing users to customize the generated audio style according to their preferences or current needs.

[0172] In some embodiments of this disclosure, the temporal guidance feature is output by a trained temporal fusion model that fuses temporal features of at least one temporal dimension, and the temporal features of each temporal dimension are output by a corresponding trained temporal feature extraction model.

[0173] In this embodiment, the temporal feature extraction model with different temporal dimensions is used to extract features of different temporal dimensions (such as rhythm, melody, etc.) from sample music waveform images. After training, it can extract the required temporal features more accurately. The trained temporal fusion model can fuse temporal features of multiple temporal dimensions, and the output can guide the performance of the generated audio in the time series during audio reconstruction.

[0174] Figure 8 This is a schematic diagram of a model processing framework in an audio generation method according to some embodiments of this disclosure. Figure 8 As shown, the framework may include a temporal feature extraction model 801, a temporal feature extraction model 802, a temporal fusion model 803, and an audio generation model 804.

[0175] refer to Figure 8First, the source music audio, text representing the music type, and waveform image representing the audio signal are determined. Then, audio noise features are determined based on the source music audio, and text-dimensional features are extracted from the text to obtain global guidance features. Temporal feature extraction models 801 and 802 are used to extract corresponding temporal dimensions from the waveform image, respectively, yielding first and second temporal features. These are then fused using a temporal fusion model 803 to obtain temporal guidance features. Finally, the audio noise features, global guidance features, and temporal guidance features are input into an audio generation model 804, which reconstructs the audio based on these features to generate the target audio.

[0176] In some embodiments of this disclosure, the audio generation method may further include training multiple initial temporal feature extraction models, an initial temporal fusion model, and the initial audio generation model in the following manner.

[0177] The process involves identifying sample music audio, sample music description text, sample music waveform image, and tag audio that matches the sample music audio, sample music description text, and sample music waveform image; determining sample noise features based on the sample music audio; extracting text-dimensional features based on the sample music description text to obtain global guidance features; extracting features from the sample music waveform image in multiple time-series dimensions using multiple initial temporal feature extraction models to obtain sample temporal features in multiple time-series dimensions; fusing the sample temporal features in multiple time-series dimensions using an initial temporal fusion model to obtain sample temporal guidance features; processing the global guidance features, the sample temporal guidance features, and the sample noise features using an initial audio generation model to perform audio reconstruction and generate predicted audio; constructing loss functions based on the predicted audio and the tag audio in the text dimension and the multiple time-series dimensions; and using the loss functions to train the multiple initial temporal feature extraction models, the initial temporal fusion model, and the initial audio generation model to obtain the trained temporal fusion model, the trained temporal feature extraction model, and the trained audio generation model.

[0178] In this embodiment of the disclosure, the sample music audio can provide the original audio information, the sample music description text can semantically describe the characteristics of the music, the sample music waveform image can visually represent the characteristics of the audio, and the labeled audio can be used as a reference standard to evaluate the quality of the generated predicted audio.

[0179] The sample music description text can be obtained by processing the sample music audio using a multimodal recognition model. The sample music waveform image can be a waveform diagram obtained by processing the sample music audio using professional tools, or it can be a dynamic curve of the sample music audio drawn by a user. For example, the dynamic curve corresponding to a certain duration (such as 10 seconds) in the waveform diagram can be drawn as a {linear, Tanh, cosine} function, which can be vertically flipped or not, and its scaling dynamic range can be controlled to {±3, ±6, ±9, ±12, ±15, ±18} decibels. The tagged audio can be produced by professional music creators.

[0180] Loss functions can be constructed based on the predicted and labeled audio in both textual and temporal dimensions. These functions can include losses for pitch, loudness, and rhythm. The loss function measures the difference between the predicted and labeled audio, considering multiple aspects of both textual and temporal dimensions to comprehensively evaluate the quality of the generated audio. For example, a multimodal recognition model can be used to process labeled audio, obtaining the labeled music description text. A loss function can then be constructed in the textual dimension based on the textual similarity between the sample music description text and the labeled music description text.

[0181] Among these methods, the parameters of the model can be continuously adjusted through optimization algorithms (such as gradient descent) to minimize the value of the loss function, ultimately resulting in a trained temporal fusion model, a trained temporal feature extraction model, and a trained audio generation model, enabling these models to better perform their respective tasks.

[0182] In this embodiment of the disclosure, after multiple iterations of training, a trained temporal fusion model, a trained temporal feature extraction model, and a trained audio generation model can be obtained simultaneously. Combined with... Figure 8 During the training process described above, the temporal feature extraction model 801, the temporal feature extraction model 802, the temporal fusion model 803, and the audio generation model 804 can be jointly trained.

[0183] Through the embodiments disclosed herein, a temporal feature extraction model, a temporal fusion model, and an audio generation model can be trained simultaneously. These models cooperate and optimize each other during training, ensuring better collaboration at each stage and improving the overall performance of the audio generation system. This results in generated audio that more closely resembles the labeled audio, enhancing the quality and accuracy of audio generation. Furthermore, by using loss functions constructed across textual and multiple temporal dimensions, the differences between the generated audio and the labeled audio can be evaluated from multiple perspectives, avoiding the limitations of single-dimensional evaluation. This allows the model to be optimized in multiple aspects, thereby better adapting to the generation needs of different types of music.

[0184] Figure 9This is a flowchart illustrating the determination of audio noise features in an audio generation method according to some embodiments of this disclosure. For example... Figure 9 As shown, in some embodiments of this disclosure, determining audio noise characteristics based on the source music audio may include the following steps.

[0185] Step S910: Perform emotion recognition on the source music audio to obtain the emotional characteristics of the source music audio.

[0186] In this embodiment of the disclosure, machine learning or deep learning models can be used to analyze audio signals and identify the emotions or feelings contained therein. For example, the model may be able to distinguish different emotional states such as happiness, sadness, anger, and calmness. The result of emotion recognition can be an emotion feature vector, which represents the intensity and type of emotion or feeling in the audio.

[0187] Step S920: Audio encoding is performed on the source music audio to obtain the signal characteristics of the source music audio.

[0188] In this embodiment of the disclosure, the process of audio encoding of the source music audio is to convert the audio signal into a series of digital representations so that the computer can process and analyze it. Audio encoding may involve steps such as sampling, quantization, and compression, ultimately resulting in a signal feature vector, which can contain the audio's time-domain features (such as amplitude, duration, etc.) and frequency-domain features (such as frequency, spectrum, etc.).

[0189] Step S930: The emotional features and the signal features are fused and encoded to obtain the audio noise features.

[0190] In this embodiment, emotional features and signal features can be superimposed and fed into a latent space encoder (such as a VAE-Encoder) for feature compression to obtain audio noise features. These audio noise features can then be used as initial noise and fed into an audio generation model for processing.

[0191] Through the embodiments of this disclosure, emotional features and signal features can be fused into audio noise features, enabling comprehensive utilization of multi-dimensional information in the source music audio. This avoids the limitations of using emotional or signal features alone, providing more comprehensive information for subsequent audio processing and improving the accuracy and richness of the processing.

[0192] In an exemplary embodiment, regarding the audio encoding part, firstly, a pre-trained audio encoder can be used for general encoding to obtain a mathematical representation of the signal and to achieve basic modeling of the global audio signal. Secondly, an emotion classification and recognition model is used to model the emotion of the audio data. The classifier of the emotion classification model can be removed, retaining only the feature modeling part of the model. The penultimate output of the emotion classification and recognition model is merged with the third-layer output in the channel dimension to obtain feature data. This realizes the modeling of the emotional features of the original audio using the emotion classification and recognition model, allowing the emotional part of the original audio to participate in audio reconstruction in the form of features. For the features of the above two parts (i.e., signal features and emotional features), they are fed into the VAE-Encoder module (the encoding module of the variational autoencoder) through superposition for feature compression. This compresses the features of the above two parts into the dimension of the latent space and realizes the initialization of the diffusion model input, generating audio noise features to participate in audio reconstruction. This embodiment simulates the noise initialization method in the image generation process. In the subsequent diffusion model generation of target audio, the input is based on the initialization of the original audio data. This operation can ensure that the changes in audio reconstruction are based on the original audio, avoiding excessive modification that would cause the final effect to be unrelated to the original audio.

[0193] In some embodiments of this disclosure, determining the source music audio includes: determining the source music audio based on a third music audio input by a user; or, in response to a user's selection operation on music attributes, determining the source music audio from preset music audio based on the selected music attributes.

[0194] In this embodiment of the disclosure, the user can directly provide a music audio file (i.e., a third-party music audio file) as material through file upload, link sharing, or other means. Subsequent operations such as emotion recognition, audio encoding, and fusion encoding can all be performed based on this audio file provided by the user.

[0195] In this embodiment of the disclosure, one or more music audio files (i.e., preset music audio files) may be pre-stored. These preset music audio files can cover different styles, genres, moods, and other attributes, and have tag classification information. Users can specify their desired music attributes through selection operations on the interface, such as selecting a "happy" mood, a "classical" music style, or a "fast-paced" rhythm type. Based on the user's selection, the audio file that best matches these attributes can be selected from the preset music audio files as the source music audio.

[0196] This disclosure allows users to directly input their own music audio, providing them with ample autonomy and satisfying their needs for reconstructing specific audio. It enables users to personalize and process their audio, increasing user engagement and system satisfaction. Furthermore, for users without their own audio, they can select source music from preset options based on music attributes, offering a convenient and quick method. This also allows for full utilization of pre-stored music resources, enabling users to quickly find suitable materials according to their needs, thus improving usability and ease of use.

[0197] Figure 10 This describes an interactive flow for generating target audio in an audio generation method according to some embodiments of this disclosure. For example... Figure 10 As shown, the interactive process for generating the target audio may include the following steps.

[0198] Step S1010: Determine the third music audio input by the user, or determine the source music audio from the preset music audio based on the user's selection operation of music attributes; determine the source music audio based on the third music audio or the source music audio.

[0199] Step S1020: Input first data for the text dimension, and determine the text based on the first data; wherein, the first data includes at least one of the following: first music audio, music genre description text.

[0200] Step S1030: Select at least one time series dimension from among multiple candidate time series dimensions.

[0201] Step S1040: Input the corresponding second data for each time series dimension, and determine the waveform image used for feature extraction of the corresponding time series dimension based on the second data; the second data includes at least one of the following: second music audio, audio waveform diagram, curve plot image, music time series description text.

[0202] Step S1050: Output target audio, which is obtained by audio reconstruction based on the source music audio, the text, and the waveform image.

[0203] The specific implementation methods for each of the above steps have been described in detail in the embodiments of the method, and will not be elaborated here.

[0204] It should be noted that the above figures are merely illustrative representations of the processes included in methods according to some embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0205] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0206] Figure 11 This is a block diagram illustrating an audio generation apparatus according to some embodiments of the present disclosure. (Refer to...) Figure 11 The device includes: a determination unit 1101, a noise feature generation unit 1102, a guidance feature generation unit 1103, an audio generation unit 1104, a style model application unit 1105, and a training unit 1106.

[0207] The determination unit 1101 is used to determine the source music audio and music control information; the noise feature generation unit 1102 is used to determine audio noise features based on the source music audio; the guidance feature generation unit 1103 is used to perform multi-dimensional feature extraction based on the music control information to obtain global guidance features and temporal guidance features; and the audio generation unit 1104 is used to perform audio reconstruction based on the global guidance features, the temporal guidance features, and the audio noise features to generate target audio that matches the music control information.

[0208] In some embodiments of this disclosure, the music control information includes text representing the music type and waveform images representing the audio signal; wherein, the guidance feature generation unit 1103 performs multi-dimensional feature extraction based on the music control information to obtain global guidance features and temporal guidance features, including: performing text-dimensional feature extraction based on the text to obtain the global guidance features; performing feature extraction based on the waveform image at least one temporal dimension to obtain temporal features of the corresponding temporal dimension; and determining the temporal guidance features based on the temporal features of each temporal dimension.

[0209] In some embodiments of this disclosure, the determining unit 1101 determines the text by: determining the text based on first data input by the user for the text dimension; wherein the first data includes at least one of the following: first music audio, music genre description text; the music genre includes at least: music genre, music mood.

[0210] In some embodiments of this disclosure, the determining unit 1101 determines the waveform image by: in response to a user's selection operation of a time series dimension among multiple candidate time series dimensions, determining the at least one time series dimension; and determining a waveform image for feature extraction of the corresponding time series dimension based on second data input by the user for the time series dimension; wherein the second data includes at least one of the following: second music audio, audio waveform graph, curve plotting image, and music time series description text.

[0211] In some embodiments of this disclosure, the at least one temporal dimension includes a melody dimension, and the waveform image includes a first waveform image; wherein, the guided feature generation unit 1103 performs feature extraction of at least one temporal dimension based on the waveform image to obtain temporal features of the corresponding temporal dimension, including: performing frame-by-frame time-frequency conversion on the first waveform image to obtain first frequency domain data of multiple frames; mapping the frequency domain energy in each first frequency domain data to the pitch in a preset pitch set, and determining the pitch features of each frame based on the pitch; performing dimensional conversion on the pitch features of each frame based on a preset number of dimensions to obtain temporal features of the melody dimension.

[0212] In some embodiments of this disclosure, the at least one temporal dimension includes dynamic intensity, and the waveform image includes a second waveform image; wherein, the guided feature generation unit 1103 performs feature extraction of at least one temporal dimension based on the waveform image to obtain temporal features of the corresponding temporal dimension, including: performing frame-by-frame time-frequency conversion on the second waveform image to obtain second frequency domain data of multiple frames; performing smooth summation on the frequency domain energy related to each second frequency domain data to obtain a smooth summation result, mapping the smooth summation result to a decibel value; and determining the temporal features of dynamic intensity based on the decibel value corresponding to each frame.

[0213] In some embodiments of this disclosure, the at least one temporal dimension includes a beat dimension, and the waveform image includes a third waveform image; wherein, the guided feature generation unit 1103 performs feature extraction of at least one temporal dimension based on the waveform image to obtain temporal features of the corresponding temporal dimension, including: performing frame-by-frame time-frequency conversion on the third waveform image to obtain third frequency domain data of multiple frames; performing beat detection on each third frequency domain data to obtain beat detection results, wherein the beat detection results are one of the following: beat points and non-beat points; and determining the temporal features of the beat dimension based on the beat detection results corresponding to each frame.

[0214] In some embodiments of this disclosure, the guidance feature generation unit 1103 determines the temporal guidance feature based on the temporal features of each temporal dimension, including: determining the dimensional order of the at least one temporal dimension; sequentially performing incremental compression and superposition on the temporal features of each temporal dimension according to the dimensional order to obtain at least one superimposed feature; wherein the number of superimposed features is the same as the number of temporal dimension types; and fusing the at least one superimposed feature to obtain the temporal guidance feature.

[0215] In some embodiments of this disclosure, the guiding feature generation unit 1103 determines the dimensional order of the at least one temporal dimension, including: determining the dimensional order of the at least one temporal dimension according to a preset dimensional priority; and / or, in response to receiving a user's sorting operation on the at least one temporal dimension, determining the dimensional order of the at least one temporal dimension.

[0216] In some embodiments of this disclosure, the audio generation unit 1104 reconstructs audio based on the global guidance features, the temporal guidance features, and the audio noise features to generate target audio that matches the music control information. This includes: inputting the global guidance features, the temporal guidance features, and the audio noise features into a trained audio generation model; wherein the trained audio generation model includes a backbone generation model and a control network, the backbone generation model being constructed based on a preset diffusion model; processing the global guidance features and the temporal guidance features through the control network; processing the audio noise features through the backbone generation model and adding guidance information to the processing of the backbone generation model using the output of the control network; and generating the target audio based on the reconstructed audio features output by the backbone generation model.

[0217] In some embodiments of this disclosure, the audio generation apparatus further includes a style model application unit 1105, configured to: determine a target music style model; the target music style model is obtained by adjusting and training the backbone generation model using music audio corresponding to the target music style; and update the backbone generation model according to the network parameters in the music style model.

[0218] In some embodiments of this disclosure, the style model application unit 1105 determines a target music style model, including: in response to the existence of a music style model associated with the user, determining the music style model as the target music style model; or, in response to the user's selection operation of a target music style among multiple candidate music styles, determining the target music style model.

[0219] In some embodiments of this disclosure, the temporal guidance feature is output by fusing temporal features of at least one temporal dimension using a trained temporal fusion model, and the temporal features of each temporal dimension are output by a corresponding trained temporal feature extraction model; the audio generation device further includes a training unit 1106; wherein, the determining unit 1101 is further configured to: determine sample music audio, sample music description text, sample music waveform image, and tag audio matching the sample music audio, sample music description text, and sample music waveform image; the noise feature generation unit 1102 is further configured to determine sample noise features based on the sample music audio; the guidance feature generation unit 1103 is further configured to perform text-dimensional feature extraction based on the sample music description text to obtain global guidance features for the sample; the guidance feature generation unit 1103 is further configured to use multiple initial times The sequence feature extraction model extracts features from the sample music waveform image in multiple temporal dimensions to obtain sample temporal features in multiple temporal dimensions. An initial temporal fusion model then fuses these multiple temporal features to obtain sample temporal guidance features. The audio generation unit 1104 is further configured to: process the sample global guidance features, the sample temporal guidance features, and the sample noise features using the initial audio generation model to perform audio reconstruction and generate predicted audio. The training unit 1106 is configured to: construct a loss function based on the predicted audio and the labeled audio in the text dimension and the multiple temporal dimensions; use the loss function to train the multiple initial temporal feature extraction models, the initial temporal fusion model, and the initial audio generation model to obtain the trained temporal fusion model, the trained temporal feature extraction model, and the trained audio generation model.

[0220] In some embodiments of this disclosure, the noise feature generation unit 1102 determines audio noise features based on the source music audio, including: performing emotion recognition on the source music audio to obtain the emotion features of the source music audio; performing audio encoding on the source music audio to obtain the signal features of the source music audio; and performing fusion encoding based on the emotion features and the signal features to obtain the audio noise features.

[0221] In some embodiments of this disclosure, the determining unit 1101 determines the source music audio by: determining the source music audio based on a third music audio input by the user; or, in response to the user's selection operation of music attributes, determining the source music audio from preset music audio based on the selected music attributes.

[0222] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0223] Figure 12 This is a block diagram illustrating an audio generation apparatus 1200 according to some embodiments of the present disclosure. For example, apparatus 1200 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0224] Reference Figure 12 The device 1200 may include one or more of the following components: a processing component 1202, a memory 1204, a power component 1206, a multimedia component 1208, an audio component 1210, an input / output (I / O) interface 1212, a sensor component 1214, and a communication component 1216.

[0225] Processing component 1202 typically controls the overall operation of device 1200, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1202 may include one or more processors 1220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1202 may include one or more modules to facilitate interaction between processing component 1202 and other components. For example, processing component 1202 may include a multimedia module to facilitate interaction between multimedia component 1208 and processing component 1202.

[0226] Memory 1204 is configured to store various types of data to support the operation of device 1200. Examples of such data include instructions for any application or method operating on device 1200, contact data, phonebook data, messages, pictures, videos, etc. Memory 1204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0227] The power supply component 1206 provides power to the various components of the device 1200. The power supply component 1206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 1200.

[0228] Multimedia component 1208 includes a screen that provides an output interface between the device 1200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1208 includes a front-facing camera and / or a rear-facing camera. When the device 1200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0229] Audio component 1210 is configured to output and / or input audio signals. For example, audio component 1210 includes a microphone (MIC) configured to receive external audio signals when device 1200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1204 or transmitted via communication component 1216. In some embodiments, audio component 1210 also includes a speaker for outputting audio signals.

[0230] I / O interface 1212 provides an interface between processing component 1202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0231] Sensor assembly 1214 includes one or more sensors for providing status assessments of various aspects of device 1200. For example, sensor assembly 1214 may detect the on / off state of device 1200, the relative positioning of components such as the display and keypad of device 1200, changes in the position of device 1200 or a component of device 1200, the presence or absence of user contact with device 1200, the orientation or acceleration / deceleration of device 1200, and temperature changes of device 1200. Sensor assembly 1214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0232] Communication component 1216 is configured to facilitate wired or wireless communication between device 1200 and other devices. Device 1200 can access wireless networks based on communication standards, such as WiFi, 3G, 4G, 5G, other communication standards, or combinations thereof. In some embodiments of this disclosure, communication component 1216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In some embodiments of this disclosure, communication component 1216 further includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0233] In some embodiments of this disclosure, the apparatus 1200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0234] In some embodiments of this disclosure, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1204 including instructions, which can be executed by a processor 1220 of the device 1200 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0235] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform an audio generation method, the method comprising: determining source music audio and music control information; determining audio noise features based on the source music audio; performing multi-dimensional feature extraction based on the music control information to obtain global guidance features and temporal guidance features; and performing audio reconstruction based on the global guidance features, the temporal guidance features, and the audio noise features to generate a target audio that matches the music control information.

[0236] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0237] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An audio generation method, characterized in that, include: Determine the source music audio and music control information; Determine the audio noise characteristics based on the provided music audio; Based on the music control information, multi-dimensional feature extraction is performed to obtain global guidance features and temporal guidance features; Audio reconstruction is performed based on the global guidance features, the temporal guidance features, and the audio noise features to generate target audio that matches the music control information.

2. The method according to claim 1, characterized in that, The music control information includes text representing the music type and waveform images representing the audio signal; Specifically, multi-dimensional feature extraction is performed based on the music control information to obtain global guidance features and temporal guidance features, including: Based on the text, text-dimensional features are extracted to obtain the global guidance features; Based on the waveform image, feature extraction is performed in at least one time-series dimension to obtain the time-series features of the corresponding time-series dimension; The timing guidance features are determined based on the timing characteristics of each timing dimension.

3. The method according to claim 2, characterized in that, Determining the text includes: The text is determined based on the first data input by the user for the text dimension; The first data includes at least one of the following: first music audio, music genre description text; the music genre includes at least: music genre, music mood.

4. The method according to claim 2, characterized in that, Determining the waveform image includes: In response to a user's selection operation of a time series dimension from multiple candidate time series dimensions, at least one time series dimension is determined; Based on the second data input by the user for the time series dimension, determine the waveform image used for feature extraction of the corresponding time series dimension; The second data includes at least one of the following: second music audio, audio waveform graph, curve plot image, and music timing description text.

5. The method according to claim 2, characterized in that, The at least one temporal dimension includes a melody dimension, and the waveform image includes a first waveform image; wherein, feature extraction of at least one temporal dimension is performed based on the waveform image to obtain temporal features of the corresponding temporal dimension, including: The first waveform image is subjected to frame-by-frame time-frequency conversion to obtain the first frequency domain data of multiple frames; The frequency domain energy in each first frequency domain data is mapped to the pitch in a preset pitch set, and the pitch feature of each frame is determined based on the pitch. Based on a preset number of dimensions, the pitch features of each frame are transformed to obtain the temporal features of the melody dimension.

6. The method according to claim 2, characterized in that, The at least one temporal dimension includes dynamic intensity, and the waveform image includes a second waveform image; wherein, feature extraction of at least one temporal dimension is performed based on the waveform image to obtain temporal features of the corresponding temporal dimension, including: The second waveform image is subjected to frame-by-frame time-frequency conversion to obtain second frequency domain data of multiple frames; The frequency domain energy associated with each second frequency domain data is smoothly summed to obtain a smooth summation result, which is then mapped to a decibel value. The temporal characteristics of dynamic intensity are determined based on the decibel values ​​corresponding to each frame.

7. The method according to claim 2, characterized in that, The at least one temporal dimension includes a beat dimension, and the waveform image includes a third waveform image; wherein, feature extraction of at least one temporal dimension is performed based on the waveform image to obtain temporal features of the corresponding temporal dimension, including: The third waveform image is subjected to frame-by-frame time-frequency conversion to obtain third frequency domain data of multiple frames; Beat detection is performed on each third frequency domain data to obtain a beat detection result, which is one of the following: a beat point and a non-beat point; The temporal features of the beat dimension are determined based on the beat detection results corresponding to each frame.

8. The method according to claim 2, characterized in that, The timing guidance features are determined based on the timing features of each timing dimension, including: Determine the dimensional order of the at least one temporal dimension; The temporal features of each temporal dimension are sequentially compressed and superimposed according to the dimensional order to obtain at least one superimposed feature; wherein, the number of superimposed features is the same as the number of temporal dimension types; The timing guidance feature is obtained by fusing the at least one superimposed feature.

9. The method according to claim 8, characterized in that, Determining the dimensional order of the at least one temporal dimension includes: The dimensional order of the at least one temporal dimension is determined according to a preset dimensional priority; and / or, In response to receiving a sorting operation from a user for the at least one time-series dimension, the dimensional order of the at least one time-series dimension is determined.

10. The method according to claim 1, characterized in that, Audio reconstruction is performed based on the global guidance features, the temporal guidance features, and the audio noise features to generate target audio that matches the music control information, including: The global guidance features, the temporal guidance features, and the audio noise features are input into the trained audio generation model; wherein, the trained audio generation model includes a backbone generation model and a control network, and the backbone generation model is constructed based on a preset diffusion model; The global guidance features and the temporal guidance features are processed through the control network; The audio noise features are processed by the backbone generation model, and the output of the control network is used to add guiding information to the processing of the backbone generation model. The target audio is generated based on the reconstructed audio features output by the backbone generation model.

11. The method according to claim 10, characterized in that, The method further includes: A target music style model is determined; the target music style model is obtained by adjusting and training the backbone generation model using music audio corresponding to the target music style. The backbone generation model is updated based on the network parameters in the music style model.

12. The method according to claim 11, characterized in that, Determine the target music style model, including: In response to the existence of a music style model associated with the user, the music style model is determined as the target music style model; or... In response to the user's selection of a target music style from multiple candidate music styles, the target music style model is determined.

13. The method according to claim 10, characterized in that, The time-series guided features are output by the trained time-series fusion model by fusing time-series features of at least one time-series dimension. The time-series features of each time-series dimension are output by the corresponding trained time-series feature extraction model. The method further includes: Identify the sample music audio, sample music description text, sample music waveform image, and the tag audio that matches the sample music audio, sample music description text, and sample music waveform image; Determine the noise characteristics of the samples based on the sample music audio; Based on the sample music description text, text-dimensional features are extracted to obtain the sample global guidance features; Multiple initial temporal feature extraction models are used to extract features from the sample music waveform image in multiple temporal dimensions to obtain sample temporal features in multiple temporal dimensions; the sample temporal features in multiple temporal dimensions are then fused using an initial temporal fusion model to obtain sample temporal guidance features. The initial audio generation model processes the global guidance features, temporal guidance features, and noise features of the samples to reconstruct the audio and generate predicted audio. Based on the predicted audio and the labeled audio, a loss function is constructed in the text dimension and the multiple temporal dimensions. The loss function is used to train the multiple initial temporal feature extraction models, the initial temporal fusion model and the initial audio generation model, to obtain the trained temporal fusion model, the trained temporal feature extraction model and the trained audio generation model.

14. The method according to claim 1, characterized in that, Determining audio noise characteristics based on the provided music audio material includes: Emotion recognition is performed on the source music audio to obtain the emotional characteristics of the source music audio; The source music audio is encoded to obtain the signal characteristics of the source music audio; The audio noise features are obtained by fusing and encoding the emotional features and the signal features.

15. The method according to claim 1, characterized in that, Determine the source music audio, including: The source music audio is determined based on a third music audio input by the user; or, In response to the user's selection of music attributes, the source music audio is determined from preset music audio based on the selected music attributes.

16. An audio generation apparatus, characterized in that, include: The determining unit is used to determine the source music audio and music control information; A noise feature generation unit is used to determine audio noise features based on the source music audio. The guidance feature generation unit is used to perform multi-dimensional feature extraction based on the music control information to obtain global guidance features and temporal guidance features; An audio generation unit is used to reconstruct audio based on the global guidance features, the temporal guidance features, and the audio noise features, and generate target audio that matches the music control information.

17. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the steps of the method according to any one of claims 1-15.

18. A non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of a mobile terminal, enable the mobile terminal to perform the steps of the method according to any one of claims 1-15.

19. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-15.

Images