Audio representation model training method, audio generation method and related product

Abstract

The embodiment of the invention discloses an audio representation model training method, an audio generation method and a related product, and the training method comprises the steps: training an audio representation model through employing annotation data of multi-party audios including a semantic text, a melody, an event to which the audio belongs, and the like; the audio representation model can be promoted to learn the matching relation between the text information of the audio and the melody and the matching relation between the text information of the audio and the event to which the audio belongs, it is guaranteed that the predicted audio representation extracts or summarizes the audio information in a diversified and comprehensive mode, and therefore generation control over the needed audio can be well achieved in the follow-up process conveniently. Besides, through the preorder audio representation predicted by the audio representation model and the audio descriptor playing a prompting role, the audio prediction model is prompted to learn the pairing relationship between the text information of the audio and the melody and the audio belonging event, so that the audio representation of the to-be-tested audio can be predicted through the input text; therefore, the generation control of the required audio can be realized conveniently, for example, songs with accurate difference between the main songs and the auxiliary songs can be generated conveniently.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of audio technology, and in particular to a training method for an audio representation model, an audio generation method, and related products. Background Art

[0002] Nowadays, more and more people like to adapt audio to generate target audio that meets their expectations.

[0003] However, current audio models represent audio signals in an average manner and are unable to identify the differences between various events in the input audio (such as the verse, chorus, bird calls, drum beats, etc.); in other words, current audio models believe that the verse and chorus are the same, and do not know what is called the verse and chorus, nor do they know the distribution of the verse and chorus in this audio. Therefore, the new audio generated does not have ideal control over the verse and chorus, which affects the user experience. Summary of the Invention

[0004] The embodiments of the present application provide a training method for an audio representation model, an audio generation method, and related products for conveniently and effectively improving the control effect of events in target audio.

[0005] A first aspect of an embodiment of the present application provides a method for training an audio representation model, comprising:

[0006] Obtaining a first sample audio and annotation data of the first sample audio; the annotation data including semantic text, melody, and audio event annotated for audio content in the first sample audio;

[0007] Extracting features from the first sample audio using the audio representation model to be trained to obtain a first audio representation; the first audio representation includes an audio representation of a semantic dimension, an audio representation of a melodic dimension, and an audio representation of a dimension of an event to which the audio belongs;

[0008] Generate predicted semantic text, predicted melody, and predicted audio event based on the first audio representation;

[0009] Determine a first prediction loss based on the predicted semantic text, the predicted melody, the event to which the predicted audio belongs, and the labeled data;

[0010] Parameters of the audio representation model are optimized based on the first prediction loss until a convergence condition is reached, thereby obtaining a trained target audio representation model.

[0011] Optionally, the process of obtaining the labeled data of the first sample audio includes:

[0012] Each of the annotated data is marked with a timestamp, so that the audio representation model can learn the pairing relationship between each word in the semantic text and the respective melody and audio event through the timestamp.

[0013] Optionally, the process of constructing the audio representation model includes:

[0014] A vector quantizer is added to the encoder of the audio representation model, so that the vector quantizer maps the vector representation output by the encoder into a discrete codebook with limited entries, thereby reducing the complexity of the audio representation model.

[0015] Optionally, the process of constructing the audio representation model includes:

[0016] Position embedding information is added before and / or after the encoder of the audio representation model to enable the audio representation model to enhance the learning of audio time information; the position embedding information includes the temporal relationship of each first sample audio and / or the content of the first sample audio input into the audio representation model.

[0017] Optionally, the training method further includes:

[0018] The representation compression amount predicted by the audio representation model is set to be smaller than a preset amount; the representation compression amount represents the number of audio representations corresponding to audio per unit time.

[0019] Optionally, after obtaining the trained target audio representation model, the training method further includes:

[0020] Optimizing parameters of the initial prediction model using the audio description of the second sample audio and the second audio representation extracted from the second sample audio by the target audio representation model until a trained audio prediction model is obtained;

[0021] The audio prediction model is used to predict the audio representation corresponding to the subsequent audio content in the audio to be tested based on the preceding semantic text and audio description language of the audio to be tested; the subsequent audio content includes a predicted semantic text that appears later than the preceding semantic text in the playback time dimension.

[0022] Optionally, the using the audio description of the second sample audio and the second audio representation extracted from the second sample audio by the trained audio representation model to optimize parameters of the initial prediction model includes:

[0023] An autoregressive model is selected as the initial prediction model, and other audio representations of the second audio sample are predicted one by one based on the autoregressive principle of the autoregressive model, the audio description of the second audio sample, and the second audio representation; the other audio representations appear later than the second audio representation in the playback time dimension;

[0024] Determining a second prediction loss; the second prediction loss is the difference between the predicted semantic text, predicted melody, and predicted audio event generated based on the other audio representation and the true semantic text, true melody, and true audio event respectively;

[0025] Parameters of the autoregressive model are adjusted based on the second prediction loss to obtain a trained autoregressive model as an audio prediction model.

[0026] When the method described in the first aspect of this application is specifically implemented, reference can be made to the contents described in the second aspect of this application.

[0027] A second aspect of the embodiments of the present application provides an audio generation method, including:

[0028] Input the preamble semantic text of the audio to be tested into the target audio representation model to obtain the preamble audio representation predicted by the target audio representation model for the preamble audio content of the audio to be tested; wherein the preamble semantic text is the text that appears before the audio to be tested ends playing, and the target audio representation model is trained according to the training method described in the first aspect or any specific implementation of the first aspect; the preamble audio content includes the preamble semantic text of the semantic dimension, and the melody dimension content and the event dimension content to which the audio belongs corresponding to the preamble semantic text in the audio to be tested;

[0029] The preceding audio representation is used to combine with the audio description of the audio to be tested to predict the subsequent audio representation corresponding to the subsequent audio content of the audio to be tested in the semantic dimension, the melodic dimension, and the audio event dimension respectively;

[0030] The subsequent audio representation is decoded and restored to obtain target audio that conforms to the audio description language.

[0031] A third aspect of an embodiment of the present application provides an electronic device, including: a processor and a memory;

[0032] The processor is configured to communicate with the memory and execute instructions in the memory to implement the method described in the first aspect of the embodiment of the present application or any specific implementation of the first aspect.

[0033] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, they implement the method described in the first aspect of the embodiment of the present application or any specific implementation of the first aspect.

[0034] A fifth aspect of an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a processor, they implement the method described in the first aspect of the embodiment of the present application or any specific implementation of the first aspect.

[0035] It can be seen from the above technical solutions that the embodiments of the present application have at least the following advantages:

[0036] The embodiment of the present application uses labeled data containing multiple audio information such as semantic text, melody, and audio events to train the audio representation model, which can prompt the audio representation model to learn the pairing relationship between the text information of the audio and the melody and audio events, respectively, to ensure that the predicted audio representation extracts or summarizes the audio information in a diversified and comprehensive manner, thereby facilitating the subsequent good control of the generation of the required audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0038] It should be noted that, although the steps in the process diagrams (if any) involved in the various embodiments are drawn in sequence as indicated by the arrows, unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The order of execution of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0039] Figure 1 A schematic diagram of the system architecture of an embodiment of the present application;

[0040] Figure 2 A flowchart of a method for training an audio representation model according to an embodiment of the present application is provided;

[0041] Figure 3Schematic diagram of training data for the audio representation model according to an embodiment of the present application;

[0042] Figure 4 This is a schematic diagram of training data for the audio prediction model according to an embodiment of the present application;

[0043] Figure 5 This is a schematic diagram of the structure of the audio prediction model according to an embodiment of the present application;

[0044] Figure 6 This is another structural diagram of the audio prediction model according to an embodiment of the present application;

[0045] Figure 7 This is a flowchart of the audio generation method according to an embodiment of the present application;

[0046] Figure 8 This is another flowchart of the audio generation method according to an embodiment of the present application;

[0047] Figure 9 This is a structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0049] The terms "first," "second," "third," "fourth," and the like (if any) in the specification and claims of the present application and in the drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0050] In the following description, reference is made to “a specific embodiment” or “a specific example” and similar expressions, which describe a subset of all possible embodiments, but it is understood that “a specific embodiment” or “a specific example” may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. In the following description, the term “a plurality” refers to at least two. In some specific examples, the numerical value mentioned in this application reaches a threshold value (if any), and the former may be greater than the latter if the latter is greater than the threshold value; if “any” or “at least one” and other similar expressions are mentioned, it may specifically refer to any one of the listed examples or any combination of these examples.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0052] See also Figure 1 , Figure 1 Schematic diagram of the network equipment environment (or system architecture diagram) applicable during the implementation of this application. During the implementation of the technical solution provided in this application, the network equipment environment used includes but is not limited to a server 101, multiple computer nodes 102, 103 and 104. Of course, the number of computer nodes can also be customized, such as set to 1.

[0053] In use, server 101 can provide services for computer nodes 102, 103 and 104, and server 101 has the service request of response computer node, transmission data, verification computer node identity and other functions.Server 101 can be the server group of single network server, multiple network server composition or the cloud server consisting of a large number of mainframes or network servers based on cloud computing.Computer nodes 102, 103 and 104 are the equipment used by users, such as can be smart phones, tablet computers, notebook computers, desktop computers, PDAs, mobile internet devices (mobile internet device, MID), wearable devices (such as smart watches, smart bracelets, etc.), smart computers, smart car-mounted and other intelligent terminals.Users can use the computer node to send corresponding service request instructions so that server 101 sends corresponding data to the computer node, and then support the computer node to provide services for users.

[0054] Exemplarily, the server 101 may be loaded with the audio representation model trained in the embodiments of the present application. The user may transmit or determine the audio to be tested to the server 101 through the computer node 102, so that the audio representation model of the server 101 relies on its own prediction ability to predict the previous audio representation token (such as vector feature) corresponding to the previous audio content in the audio to be tested; in addition, the server 101 may also predict the subsequent audio representations corresponding to the subsequent audio content in the audio to be tested in the semantic dimension, melody dimension, and audio event dimension respectively through the previous audio representation and the audio description of the audio to be tested (which can be customized by the user); thereafter, the server 101 may convert the subsequent audio representation into the required audio signal through a decoding and restoration (vocoder decoder) module. It should be noted that the method provided in the embodiments of the present application may be jointly implemented by the server and the computer node as described above, or may be all implemented on the server side, or may also be all implemented on the computer node side, which may be specifically determined according to the actual application scenario and is not limited herein.

[0055] The method of the present application will be described in detail below.

[0056] Please refer to Figure 2 , a specific embodiment of a method for training an audio representation model is provided in the first aspect of the present application. The embodiment includes the following operation steps:

[0057] Step S21, obtain a first sample audio and annotation data of the first sample audio;

[0058] The annotation data includes semantic text, melody, and audio event annotated for the audio content in the first sample audio. The reason for text annotation here is that in actual situations, audio files (i.e., the first sample audio) generally are audio signals without text information and need special processing to extract text information such as lyrics.

[0059] The above audio content may specifically be at least one of lyrics and background sounds. For example, when the audio content is lyrics, the annotation data may be the specific lyrics text, the melody information (such as at least one of pitch, rhythm, and length) of each word segmentation in the lyrics text in the audio, and the audio event corresponding to each word segmentation in the audio (such as song structures like the main chorus); when the audio content is background music such as bird calls or instrument sounds, the annotation data may be the onomatopoeia for this background music (such as yingying, dongdongqiang, etc.), the melody information of this background music in the audio, and the corresponding audio event (such as belonging to events like bird calls, drum sounds, etc.). The aforementioned audio event may be referred to as a structure instruction or an event instruction, and this kind of instruction similar to an audio event is introduced into the audio representation model. For details, please refer to the following text.

[0060] It should be noted that the aforementioned song structure refers to the organizational form followed in a song's creation and arrangement. It determines how the various parts of the song are arranged and connected to ensure coherence and appeal in terms of emotional and storytelling expression. A typical song structure usually includes the following sections: intro, verse, pre-chorus, chorus, bridge, and outro. The main differences in melody and arrangement between the chorus and verse are: the verse has a smooth and simple melody and a relatively simple arrangement, which serves to build emotional foundation and narrate the story; while the chorus has a high, repetitive melody, rich and varied arrangements, and a tight rhythm, aiming to create an emotional climax and enhance the song's appeal. This requires that these paragraph-by-paragraph structural differences be modeled in the audio representation.

[0061] Step S22: extracting features from the first sample audio using the audio representation model to be trained to obtain a first audio representation;

[0062] The first audio representation includes a semantic audio representation, a melodic audio representation, and an audio representation of the event to which the audio belongs. The audio representation may refer to processing and transforming an audio signal using specific technical means to extract information representing the audio characteristics. This audio representation (e.g., a token) may exist in the form of a vector or scalar.

[0063] Step S23: generating a predicted semantic text, a predicted melody, and an event to which the predicted audio belongs based on the first audio representation, and determining a first prediction loss based on the predicted semantic text, the predicted melody, the event to which the predicted audio belongs, and the annotated data;

[0064] The above-mentioned first prediction loss is the gap between the description of the first sample audio by the audio representation (i.e., the predicted token) predicted by the audio representation model in terms of semantic text, melody, and the event to which the audio belongs and the true description (i.e., the true token). This prediction loss can be mean square error loss and / or cross entropy loss, etc. In contrast, in order to enhance the prediction accuracy of the model, the specific description of the above-mentioned annotated data can be at least partially a true description, that is, some information descriptions in the annotated data can be incorrect, which can be determined by the actual situation and is not limited here. In addition, in some examples, the audio representation of at least part of the audio content in the audio (such as the lyrics after a short paragraph of lyrics) can be predicted, which can be set as needed and is not limited here.

[0065] Step S24: Optimize the parameters of the audio representation model based on the first prediction loss until a convergence condition is reached, thereby obtaining a trained target audio representation model.

[0066] In summary, the embodiment of the present application uses labeled data containing multiple audio information such as semantic text, melody, and audio events to train the audio representation model, which can enable the audio representation model to learn the pairing relationship between the text information of the audio and the melody and audio events, respectively, to ensure that the predicted audio representation extracts or summarizes the audio information in a diversified and comprehensive manner, thereby facilitating the subsequent good control of the generation of the required audio.

[0067] Based on the above examples, the method of the present application will be further described in detail below, and some specific possible implementation examples will be provided. In actual applications, the implementation contents between these examples can be combined or implemented separately as needed according to the corresponding functional principles and application logic. If implemented in combination, the execution order between the combined examples can be determined according to their respective processing logic, which can be determined by the actual scenario.

[0068] In order to make the model learn the pairing relationship between the labeled data of a certain audio more accurately, based on Figure 2 In some specific examples, the process of step S21 may include (introducing time information): marking each annotated data with a timestamp so that the audio representation model can learn the pairing relationship between each word in the semantic text and its respective melody and audio event through the timestamp.

[0069] like Figure 3 As shown, the training data for the audio representation model can include audio files and annotated data pairs. This annotated data pair includes lyrics annotations for the audio file (such as words labeled as abcde), pitch annotations, and musical structure annotations (such as verse tags). These annotations are all timestamp-equivalent (can be considered time series data), and this time corresponds to the actual content of the audio. For example, this timestamp information can be the word-by-word timestamp of each word and its pitch in the lyrics, specifically including the time point at which each word and its pitch begins to appear in the entire audio and the duration of each occurrence. Introducing time information in this way allows the model to know that the word "a" corresponds to the pitch B3 and the word "b" corresponds to the pitch E4, enhancing the model's ability to model information correspondences.

[0070] based on Figure 2 In some specific examples, the construction process of the above-mentioned audio representation model includes (design of the internal structure of the model): adding a vector quantizer to the encoder of the audio representation model, so that the vector quantizer maps the vector representation output by the encoder to a discrete codebook with limited entries, thereby reducing the complexity of the audio representation model.

[0071] like Figure 5As shown in Figure 1, the audio representation model can be a self-supervised representation model that integrates speech recognition (ASR), music event detection (MED), and singing to musical notation (STT) tasks to generate supervised music representations. The task is specified by inputting a series of specific input tokens into the decoder, for example, <asr> 、 <med> 、 <stt>These tags can be used to separately specify whether to perform tasks such as predicting semantic text (content corresponding to audio), music events, and melodies. In addition, this representation base model can be specifically trained through mask learning, and the loss is the cross-entropy loss between the predicted token and the true input token. Supplementary note: Figure 5 The numbers 60 and 62 shown can refer to the duration of a single word segment (such as the character "今").

[0072] For another example Figure 6 As shown, a vector quantizer (VQ, Vector Quantizer) can be added after a certain layer of the encoder of the audio representation model to implement a downstream small task of ASR, strengthening the accuracy of the model for speech recognition, such as enabling more accurate recognition of semantic content such as the lyric text of the input audio P(Y丨X), where X is the input audio and Y is the text content recognized from the audio, and decoder(ASR) represents the decoder specifically for speech recognition tasks; the combination of encoder1 and the vector quantizer can be regarded as a tokenzier for generating audio representation tokens that are mainly used in subsequent music generation tasks.

[0073] The vector quantizer can use a single codebook containing a finite number of entries (such as 4096 entries) (or multiple-layer codebooks can be used) to discretize the output vector obtained by the encoder's transformation of audio features. Specifically, the encoder is to find and output the discrete vector that is most similar to the input continuous vector, while the vector quantizer discretizes the representation of the continuous or discrete input vector, that is, maps the input vector to the closest discrete codebook (the codebook contains a set of finite discrete vectors), playing a role of discretization, so as to output the discretized audio representation token (which can be called the discretized feature) to reduce the model complexity.

[0074] Based on Figure 2 In some specific examples, the construction process of the above audio representation model includes (introducing position embedding information): adding position embedding information before and / or after the encoder of the audio representation model to enable the audio representation model to enhance the learning of audio time information; the position embedding information includes the timing relationship of each first sample audio and / or the content of the first sample audio input into the audio representation model input into the audio representation model.

[0075] For example, temporal information can be enhanced by adding position embeddings (P) after quantization. Because position embeddings contain temporal information, the model's encoder can capture temporal features of the audio signal during learning, such as at least one of the timestamps of the time an audio signal enters the model and the start and end words of the lyrics, thereby improving the understanding and feature description capabilities of the audio content.

[0076] based on Figure 2 In some specific examples, the training method of the embodiment of the present application may further include: setting the representation compression amount predicted by the audio representation model to be less than a preset amount; the representation compression amount represents the number of audio representations corresponding to the audio per unit time.

[0077] For example, the audio representation sequence output by the audio representation model can be set to display a frequency of 25 Hz (i.e., 1 second of audio corresponds to 25 tokens), or it can be customized to other representation compression amounts, such as 50 Hz. Here, the smaller the representation compression amount, i.e., the number of hertz (e.g., less than the preset number of 55 Hz), the lower the difficulty of prediction for the audio generation model and the lower the computational complexity; this is because when the number of features is reduced, the number of parameters that the audio generation model needs to learn will also be reduced accordingly, which usually results in a lower model complexity and easier training.

[0078] based on Figure 2 In some specific examples, after step S24, the training method of the embodiment of the present application may further include: using the audio description of the second sample audio and the second audio representation extracted by the target audio representation model from the second sample audio, optimizing the parameters of the initial prediction model until a trained audio prediction model is obtained; the audio prediction model is used to predict the audio representation corresponding to the subsequent audio content in the audio to be tested through the preceding semantic text and the audio description of the audio to be tested; the subsequent audio content includes a predicted semantic text that appears later than the preceding semantic text in the playback time dimension.

[0079] The above audio description can be customized by the user and can refer to text descriptions such as the audio style (such as pop music or sad song), background summary (such as the sound of Christmas-related musical elements such as bells), and lyrics extraction (such as "love"). This audio description can serve as a prompt for the model to better understand and represent the audio content.

[0080] It can be seen that the audio prediction model trained as above can be used to predict the audio representation corresponding to the subsequent audio content in the audio to be tested based on the preceding semantic text (which may include at least part of the lyrics text) and audio description of the audio to be tested.

[0081] In contrast, the second sample audio and the first sample audio may be the same or different audios. For example, the training data of the audio representation model may also be used as the training data of the audio generation model. The training data requirements of the audio generation model (which may specifically refer to the above-mentioned audio prediction model) may be: there must be corresponding lyrics text and its audio description text and other semantic texts, and there must be descriptions of diversified information such as the event to which the audio belongs (such as the main chorus, whistle sound, etc.), melody information, etc. In some specific examples, unlike the training data of the representation model, the training data of the audio generation model may not carry timestamp information similar to the above-mentioned timestamp information (such as Figure 4 There is no time dividing line between the lyrics shown). This is because if the audio generation model uses autoregressive reasoning, it will predict the length of time the lyrics need to be sung without the need to introduce additional time information to avoid interfering with the prediction effect.

[0082] Of course, as another possible implementation method, the training data of the audio generation model can also contain time information. In this case, semantic text with time information needs to be input during model inference, so that the autoregressive reasoning method can be avoided, which is a bit troublesome.

[0083] In some specific examples, the process of optimizing the parameters of the initial prediction model using the audio description of the second sample audio and the second audio representation extracted from the second sample audio by the trained audio representation model may include: selecting an autoregressive model as the initial prediction model, and predicting other audio representations of the second sample audio (such as Figure 8 The other audio representations appear later than the second audio representation in the playback time dimension; a second prediction loss is determined; the second prediction loss is the difference between the predicted semantic text, predicted melody and predicted audio event generated based on the other audio representations and the real semantic text, real melody and real audio event respectively; the parameters of the autoregressive model are adjusted based on the second prediction loss to obtain a trained autoregressive model as the audio prediction model.

[0084] Specifically, the initial prediction model can be a large language LLM network (also known as an autoregressive model) using the autoregressive concept, such as RNN, LSTM, etc. Alternatively, a model other than the autoregressive model can be used, such as a diffusion model, a hybrid model combining an autoregressive model and a non-autoregressive model, which can be determined by the needs. For example, the LLM network can use the mean square error (MSE) loss function to calculate the autoregressive loss (i.e., the second prediction loss). The formula for this loss can be as follows:

[0085]

[0086] Among them, y i Is the real token value, is the predicted token value, and n is the number of samples. The LLM network can be trained using the MSE loss until the MSE loss converges, and the trained LLM network can be saved.

[0087] It is to be noted that the audio representation of the embodiment of the present application serves as a bridge connecting the LLM (large language model) and the vocoder decoder (decoding and restoration) (e.g. Figure 7 ). Among them: the LLM module is the focus of the model. LLM has the ability to generate output similar to human text and can generate coherent and grammatically correct text. This creative generation ability makes LLM very useful in audio generation models. It can generate novel music works or generate specific music styles according to user instructions. And LLM can enhance the understanding of the relationship between audio and text prompts. Therefore, in the tasks of the embodiments of the present application, it is very appropriate to use LLM to model the relationship between the input text and the output audio representation. In order to model this correspondence, the audio representation predicted by LLM needs to contain information corresponding to the input text. The decoding and restoration (vocoderdecoder) module can be independent of the above-mentioned audio prediction model, or the decoding and restoration can constitute an audio generation model together with the audio prediction model: in the generative model, the role of the decoding and restoration module is to convert the intermediate representation (such as token) generated by the upstream model back to the original form such as Mel spectrum or audio signal.

[0088] As explained above, the training data of the initial prediction model includes diverse content such as the semantic text of the audio, the melody, and the event to which the audio belongs. Therefore, the audio representation finally predicted by the prediction model also includes these diverse contents and the pairing relationship between these contents, so that the prediction model can predict the subsequent audio representation sequence based on the corresponding semantic text of the previous sequence. For example, it can further ensure that the target audio finally generated based on the subsequent audio representation sequence is an ideal audio with well-controlled structures such as the main chorus.

[0089] In summary, it can be understood that the audio representation model of the embodiment of the present application is nested in the audio prediction model. Among them, the audio representation model is trained using annotated data containing multiple audio information such as semantic text, melody, and the event to which the audio belongs, so that the predicted audio representation can extract or summarize the audio information in a diversified and comprehensive manner. In addition, using the predicted audio representation (which may contain semantic content such as lyrics) and the audio description language that serves as a prompt as training data can prompt the audio prediction model to learn the pairing relationship between the text information of the audio and the melody and the event to which the audio belongs, so that the audio representation of the audio to be tested can be predicted through the preceding semantic text of the audio to be tested, thereby facilitating the generation control of the required audio, such as facilitating the generation of songs with accurate differences between the main and chorus.

[0090] Compared to Figure 2 The examples shown indicate that the above-mentioned additional or detailed example contents or possible implementation methods do not necessarily have to be executed during specific implementation. If more than two examples or possible implementation methods are added, these examples or possible implementation methods can be implemented in combination or separately. If implemented in combination, the execution order of the combined examples can be determined according to their respective processing logic, which may depend on the actual scenario.

[0091] The above mainly explains the model training phase. The following will explain the model inference or application phase in detail.

[0092] See also Figure 8 The second aspect of the present application provides a specific embodiment of an audio generation method, the operation steps of the method include:

[0093] Step S81: inputting the preamble semantic text of the audio to be tested into the target audio representation model to obtain the preamble audio representation predicted by the target audio representation model for the preamble audio content of the audio to be tested;

[0094] The preceding semantic text is the text that appears before the audio to be tested finishes playing. In other words, because this embodiment mainly predicts the audio representation of the subsequent lyrics based on the preceding lyrics, the speech text input into the target audio representation model or audio prediction model is not the last lyric text relative to the entire semantic text. This is so that there is space or demand to predict the subsequent audio representation. The preceding audio content includes the preceding semantic text in the semantic dimension, as well as the melody dimension content and the event dimension content corresponding to the preceding semantic text in the audio to be tested.

[0095] The preceding audio representation is combined with the audio description of the test audio to predict the corresponding subsequent audio representation of the test audio content in the semantic, melodic, and event dimensions. As described above, this prediction process can be implemented using an audio prediction model.

[0096] The target audio representation model is trained according to the training method described in the first aspect or any specific method embodiment of the first aspect, and the details are not repeated here. Of course, the specific implementation process of each operation in the first aspect of this application can also refer to the relevant description of the second aspect.

[0097] Step S82: Decode and restore the subsequent audio representation to obtain target audio that conforms to the audio description language.

[0098] As explained above, the audio representation model may not be involved in the model inference stage. The main purpose is for the audio prediction model to use its own learned audio representation prediction capabilities to infer the audio representation sequence. Figure 8 The model autoregressive prediction process shown in the figure, the audio prediction model (such as LLM) receives the input sequence, predicts the next corresponding audio representation token based on the input sequence, and repeats this process to generate a complete audio representation sequence, where there can be a start token to indicate the start of the prediction and an end token to indicate the end of the prediction:

[0099] Step 1: Input sequence: The model receives an initial input sequence, which is usually a string of text input (including pre-order semantic text), such as Figure 8 The text description "popular" and lyrics input "Today I stand" are shown; this string of text input is converted into an audio representation in vector form (such as a token) and spliced ​​according to the playback time dimension (to avoid ambiguity caused by incorrect context order) and fed into the audio prediction model.

[0100] Step 2: Predict the next word token: The audio prediction model predicts the next audio representation token based on the input sequence. This can be considered as predicting the next word token word by word from the previous token sequence.

[0101] Step 3: Update the sequence: Add the predicted token to the previous input sequence to form a new sequence.

[0102] Step 4. Repeat the process: return to steps 2 and 3 until the end token is encountered and the prediction stops. The audio representation sequence tokens are obtained and the prediction task is completed.

[0103] Finally, the predicted audio representation sequence tokens are fed into the decoding restoration model to convert the target audio.

[0104] In summary, the embodiments of the present application combine melody information, semantically relevant information, and some important structural information into a single audio representation. This audio representation is then used in an audio generation model. This allows the input of semantic text such as lyrics, and the structure tags learned by the model (e.g., tags belonging to the verse or chorus) to generate well-defined verse-chorus sections. This allows users with lower audio cognition or processing skills to easily create an ideal arrangement. Furthermore, compared to traditional methods, the embodiments of the present application have stronger and more obvious control over the verse-chorus structure.

[0105] See also Figure 9 The electronic device of the embodiment of the present application may include one or more processors (such as central processing units (CPU)) and a memory, in which one or more applications or data are stored.

[0106] The memory may be volatile or persistent. The program stored in the memory may include one or more modules, each of which may include a series of instruction operations on the electronic device. Furthermore, the processor may be configured to communicate with the memory and execute the series of instruction operations in the memory on the electronic device.

[0107] The electronic device may also include one or more power supplies, one or more wired or wireless network interfaces, one or more input and output interfaces, and / or one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0108] The processor can execute the operations performed by the aforementioned first aspect or any specific method embodiment of the first aspect, and the details will not be repeated here.

[0109] The present application provides a computer-readable storage medium, comprising instructions. When the instructions are executed on a computer, the computer executes the method described in the first aspect or any specific implementation of the first aspect.

[0110] The present application provides a computer program product comprising instructions or a computer program. When the computer program product is run on a computer, the computer is enabled to execute the method described in the first aspect or any specific implementation of the first aspect.

[0111] It is understood that in the various embodiments of the present application, the sequence number of each step does not mean the order of execution. The order of execution of each step should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The operation content added or refined in each example scheme of the above-mentioned method, system or device (if any) does not necessarily have to be executed in the specific implementation. If two or more operations are added, these operations can be implemented in combination or separately, depending on the actual scenario.

[0112] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system (if any) and device can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system or device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection of some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0114] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0115] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product (or computer program product) is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a business server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.< / stt> < / med> < / asr>

Claims

1. A method for training an audio representation model, characterized in that: include: Obtaining a first sample audio and labeled data of the first sample audio; The annotation data includes semantic text, melody and audio event annotated for the audio content in the first sample audio; Extracting features from the first sample audio using the audio representation model to be trained to obtain a first audio representation; the first audio representation includes an audio representation of a semantic dimension, an audio representation of a melodic dimension, and an audio representation of a dimension of an event to which the audio belongs; generating a predicted semantic text, a predicted melody, and a predicted audio event based on the first audio representation, and determining a first prediction loss based on the predicted semantic text, the predicted melody, the predicted audio event, and the annotated data; Parameters of the audio representation model are optimized based on the first prediction loss until a convergence condition is reached, thereby obtaining a trained target audio representation model.

2. The method for training an audio representation model according to claim 1, wherein: The process of obtaining the labeled data of the first sample audio includes: Each of the annotated data is marked with a timestamp, so that the audio representation model can learn the pairing relationship between each word in the semantic text and the respective melody and audio event through the timestamp.

3. The method for training an audio representation model according to claim 1, wherein: The process of constructing the audio representation model includes: A vector quantizer is added to the encoder of the audio representation model, so that the vector quantizer maps the vector representation output by the encoder into a discrete codebook with limited entries, thereby reducing the complexity of the audio representation model.

4. The method for training an audio representation model according to any one of claims 1 to 3, characterized in that: The process of constructing the audio representation model includes: Position embedding information is added before and / or after the encoder of the audio representation model to enable the audio representation model to enhance the learning of audio time information; the position embedding information includes the temporal relationship of each first sample audio and / or the content of the first sample audio input into the audio representation model.

5. The method for training an audio representation model according to claim 1, wherein: The training method further comprises: The representation compression amount predicted by the audio representation model is set to be smaller than a preset amount; the representation compression amount represents the number of audio representations corresponding to audio per unit time.

6. The method for training an audio representation model according to claim 1, wherein: After obtaining the trained target audio representation model, the training method further includes: Optimizing parameters of the initial prediction model using the audio description of the second sample audio and the second audio representation extracted from the second sample audio by the target audio representation model until a trained audio prediction model is obtained; The audio prediction model is used to predict the audio representation corresponding to the subsequent audio content in the audio to be tested based on the preceding semantic text and audio description language of the audio to be tested; the subsequent audio content includes a predicted semantic text that appears later than the preceding semantic text in the playback time dimension.

7. The method for training an audio representation model according to claim 6, wherein: The method of optimizing parameters of the initial prediction model using the audio description of the second sample audio and the second audio representation extracted from the second sample audio by the trained audio representation model comprises: An autoregressive model is selected as the initial prediction model, and other audio representations of the second audio sample are predicted one by one based on the autoregressive principle of the autoregressive model, the audio description of the second audio sample, and the second audio representation; the other audio representations appear later than the second audio representation in the playback time dimension; Determining a second prediction loss; the second prediction loss is the difference between the predicted semantic text, predicted melody, and predicted audio event generated based on the other audio representation and the true semantic text, true melody, and true audio event respectively; Parameters of the autoregressive model are adjusted based on the second prediction loss to obtain a trained autoregressive model as an audio prediction model.

8. An audio generation method, characterized in that: include: Inputting the preamble semantic text of the audio to be tested into the target audio representation model to obtain the preamble audio representation predicted by the target audio representation model for the preamble audio content of the audio to be tested; wherein the preamble semantic text is the text that appears before the audio to be tested ends playing, and the preamble audio content includes the preamble semantic text of the semantic dimension, as well as the melody dimension content and the event dimension content to which the audio belongs corresponding to the preamble semantic text in the audio to be tested; the target audio representation model is trained according to the training method according to any one of claims 1 to 7; The preceding audio representation is used to combine with the audio description of the audio to be tested to predict the subsequent audio representation corresponding to the subsequent audio content of the audio to be tested in the semantic dimension, the melodic dimension, and the audio event dimension respectively; The subsequent audio representation is decoded and restored to obtain target audio that conforms to the audio description language.

9. An electronic device, characterized in that: include: processor and memory; The processor is configured to communicate with the memory and execute instructions in the memory to implement the training method according to any one of claims 1 to 7 or the audio generation method according to claim 8.

10. A readable storage medium, characterized in that: The readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the training method according to any one of claims 1 to 7 or the audio generation method according to claim 8 is implemented.

11. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a processor, the training method according to any one of claims 1 to 7 or the audio generation method according to claim 8 is implemented.

Images