Audio generation model training method and device, and storage medium

Abstract

The invention discloses an audio generation model training method and device and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a first training data set, wherein the first training data set comprises at least one first training sample; obtaining a first logarithmic probability corresponding to the positive sample audio and a second logarithmic probability corresponding to the negative sample audio through an audio generation model according to the first training sample; obtaining a first reference logarithm probability corresponding to the positive sample audio and a second reference logarithm probability corresponding to the negative sample audio through a reference strategy model according to the first training sample; and training the audio generation model according to the first logarithm probability, the first reference logarithm probability, the second logarithm probability and the second reference logarithm probability to obtain a trained audio generation model. According to the method, by executing preference optimization training, the trained audio generation model can learn the ability of understanding and predicting the preference, and the quality of audio generation is fully improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, device, and storage medium for an audio generation model. Background Art

[0002] AI (Artificial Intelligence) technology is quietly changing the way music is created. The audio generation model is a model that generates corresponding audio based on the prompt text entered by the user. More and more users are creating music through audio generation models.

[0003] In related technologies, audio generation models typically take the form of diffusion models, which generate audio. Users input text prompts, such as lyrics, emotional descriptions, or musical styles, which guide the model's audio generation. Diffusion models process sequential data using cross-self-attention, capturing dependencies between text prompt features. These features are then decoded to generate audio that matches the prompts.

[0004] However, the audio quality (e.g., musicology, sound quality, etc.) generated by the above methods is relatively average. Summary of the Invention

[0005] The present application provides a method, device, and storage medium for training an audio generation model. The technical solutions provided by the present application are as follows:

[0006] According to one aspect of an embodiment of the present application, a method for training an audio generation model is provided, the method comprising:

[0007] Obtaining a first training data set, where the first training data set includes at least one first training sample, each first training sample including a first prompt text, at least one positive audio sample, and at least one negative audio sample, where the positive audio sample is audio that meets an audio generation condition, and the negative audio sample is audio that does not meet the audio generation condition, where the audio generation condition is a condition related to audio quality;

[0008] Obtaining, by the audio generation model and based on the first training sample, a first logarithmic probability corresponding to the positive sample audio and a second logarithmic probability corresponding to the negative sample audio, wherein the first logarithmic probability is used to represent the degree of preference of the first prompt text for the positive sample audio, and the second logarithmic probability is used to represent the degree of preference of the first prompt text for the negative sample audio;

[0009] Obtaining, based on the first training sample and using a reference strategy model, a first reference logarithmic probability corresponding to the positive sample audio and a second reference logarithmic probability corresponding to the negative sample audio;

[0010] The audio generation model is trained according to the first logarithmic probability, the first reference logarithmic probability, the second logarithmic probability, and the second reference logarithmic probability to obtain a trained audio generation model.

[0011] According to one aspect of an embodiment of the present application, a training device for an audio generation model is provided, the device comprising:

[0012] a data acquisition module, configured to acquire a first training data set, the first training data set including at least one first training sample, each first training sample including a first prompt text, at least one positive audio sample, and at least one negative audio sample, wherein the positive audio sample is audio that meets an audio generation condition, and the negative audio sample is audio that does not meet the audio generation condition, wherein the audio generation condition is a condition related to audio quality;

[0013] A first generation module is configured to obtain, using the audio generation model and the first training sample, a first logarithmic probability corresponding to the positive sample audio and a second logarithmic probability corresponding to the negative sample audio, wherein the first logarithmic probability is used to represent the preference of the first prompt text for the positive sample audio, and the second logarithmic probability is used to represent the preference of the first prompt text for the negative sample audio;

[0014] A second generating module is configured to obtain, based on the first training sample and using a reference strategy model, a first reference logarithmic probability corresponding to the positive sample audio and a second reference logarithmic probability corresponding to the negative sample audio;

[0015] A model training module is used to train the audio generation model based on the first logarithmic probability, the first reference logarithmic probability, the second logarithmic probability, and the second reference logarithmic probability to obtain a trained audio generation model.

[0016] According to one aspect of an embodiment of the present application, a computer device is provided, comprising a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned training method of the audio generation model.

[0017] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned training method of the audio generation model.

[0018] According to one aspect of an embodiment of the present application, a computer program product is provided, which includes a computer program, and the computer program is loaded and executed by a processor to implement the above-mentioned training method of the audio generation model.

[0019] The technical solutions provided in the embodiments of the present application can bring the following beneficial effects:

[0020] By using the audio generation model and the reference strategy model to predict the preference of the first prompt text for the positive sample audio and the negative sample audio respectively, the tendencies of the audio generation model and the reference strategy model in responding to the positive sample audio and the negative sample audio respectively can be obtained. Therefore, after performing preference optimization training on the audio generation model, the audio generated by the trained audio generation model can be biased towards the audio features of the positive sample audio, so that the trained audio generation model can generate audio that meets the audio generation conditions. Compared with the related art of only performing audio generation training on the audio generation model, the technical solution of the present application allows the trained audio generation model to learn the ability to understand and predict preferences, more efficiently capture the subtle differences in preferences, so that the generated audio can meet the preferences corresponding to the positive sample audio, so that the quality of the generated audio can meet the audio generation conditions, and fully improve the audio generation quality of the audio generation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a schematic diagram of an implementation environment for a solution provided by an embodiment of the present application;

[0022] Figure 2 This is a flowchart of a method for training an audio generation model provided by one embodiment of the present application;

[0023] Figure 3 This is a schematic diagram of adding marks to a lyrics sample provided by an embodiment of the present application;

[0024] Figure 4 This is a flowchart of a complete training method for an audio generation model provided by one embodiment of the present application;

[0025] Figure 5 This is a flowchart of an audio generation method based on an audio generation model provided by one embodiment of the present application;

[0026] Figure 6 This is a schematic diagram of a process for generating output audio according to an embodiment of the present application;

[0027] Figure 7 This is a schematic diagram of a process for recursively generating output audio multiple times according to an embodiment of the present application;

[0028] Figure 8This is a block diagram of a training device for an audio generation model provided by one embodiment of the present application;

[0029] Figure 9 This is a structural block diagram of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0031] Please refer to Figure 1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment can be implemented as an audio generation system. The solution implementation environment may include: a model training device 10 and a model use device 20.

[0032] The model training device 10 can be an electronic device such as a mobile phone, tablet computer, laptop computer, desktop computer, smart TV, multimedia player, vehicle terminal, server, intelligent robot, or other electronic devices with strong computing capabilities. The model training device 10 is used to train the audio generation model.

[0033] In the embodiment of the present application, the audio generation model is a machine learning model trained based on an audio generation model training method, and is used to generate output audio features corresponding to the input prompt text based on the text features of the input prompt text. The model training device 10 can use machine learning to train the audio generation model so that it has the ability to generate output audio features corresponding to the input prompt text. The specific model training method can be referred to in the following embodiment.

[0034] In an embodiment of the present application, the input data of the audio generation model includes the text features of the input prompt text and the i-th output audio features, and the output data is the i+1-th output audio features corresponding to the input prompt text.

[0035] The trained audio generation model can be deployed in model-using device 20 for use. Model-using device 20 can be an electronic device such as a mobile phone, tablet computer, laptop computer, desktop computer, smart TV, multimedia player, vehicle-mounted terminal, server, intelligent robot, or other electronic device with strong computing capabilities. When the output audio corresponding to the input prompt text is required, model-using device 20 can implement the above function using the trained audio generation model.

[0036] The model training device 10 and the model using device 20 can be two independent devices or the same device. If the model training device 10 and the model using device 20 are the same device, the model training device 10 can be deployed in the model using device 20.

[0037] In the embodiment of the present application, the execution subject of each step may be a computer device, which may be as follows: Figure 1 The model training device 10 may also be a model using device 20. The server may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0038] Please refer to Figure 2 , which shows a flow chart of a method for training an audio generation model provided by one embodiment of the present application. The execution subject of each step of the method may be a computer device. The method may include at least one of the following steps 210 to 230:

[0039] Step 210: Obtain a first training data set, where the first training data set includes at least one first training sample. Each first training sample includes a first prompt text, at least one positive sample audio, and at least one negative sample audio. The positive sample audio is audio that meets the audio generation conditions, and the negative sample audio is audio that does not meet the audio generation conditions. The audio generation conditions are conditions related to the quality of the audio.

[0040] The first prompt text includes an audio description text of the first sample audio and a lyric text of the first sample audio. The first sample audio can be any song audio, which is audio that includes an accompaniment track and a vocal track, and the vocal track includes lyric information. Optionally, the first sample audio can be any song audio available on the Internet or any song audio stored locally. Optionally, the first sample audio can be a song audio obtained by real human voice singing or a song audio obtained by mechanically synthesized human voice.

[0041] The audio description text of the first sample audio is used to describe the audio features of the first sample audio. The audio description text of the first sample audio includes at least one of the following: a genre description text of the first sample audio, a rhythm description text of the first sample audio, a tone description text of the first sample audio, a timbre description text of the first sample audio, a duration description text of the first sample audio, a sound quality description text of the first sample audio, and a musicality description text of the first sample audio. Among them, the genre description text of the first sample audio is used to describe the genre type of the first sample audio, the rhythm description text of the first sample audio is used to describe the speed of the rhythm of the first sample audio, the tone description text of the first sample audio is used to describe the pitch of the first sample audio, the timbre description text of the first sample audio is used to describe the timbre features of the first sample audio, the duration description text of the first sample audio is used to describe the audio duration of the first sample audio, the sound quality description text of the first sample audio is used to describe the audio quality of the first sample audio, and the musicality description text of the first sample audio is used to describe the musicality features of the first sample audio.

[0042] The lyric text of the first sample audio refers to the lyric text contained in the vocal track of the first sample audio, that is, the lyric text sung by the vocals in the first sample audio. The lyric text of the first sample audio contains a verse mark and a chorus mark. The verse mark is used to indicate the lyric text in the lyric text of the first sample audio that corresponds to the verse audio of the first sample audio, and the chorus mark is used to indicate the lyric text in the lyric text of the first sample audio that corresponds to the chorus audio of the first sample audio. Optionally, a verse mark can be added to each character in the lyric text corresponding to the verse audio of the first sample audio, and a chorus mark can be added to each character in the lyric text corresponding to the chorus audio of the first sample audio. You can refer to Figure 3 As shown, in Figure 3 In the figure (1), a verse mark 301 is added to each character in the lyrics text corresponding to the verse audio of the sample audio, and a chorus mark 302 is added to each character in the lyrics text corresponding to the chorus audio of the sample audio. Optionally, a verse mark can be added to the first character in the lyrics text corresponding to the verse audio of the first sample audio, and a chorus mark can be added to the first character in the lyrics text corresponding to the chorus audio of the first sample audio. Then the lyrics text between the verse mark and the chorus mark is the lyrics text corresponding to the verse audio of the first sample audio, and the lyrics text between the chorus mark and the next verse mark is the lyrics text corresponding to the chorus audio of the first sample audio. For example, in Figure 3 In Figure (2), a verse mark 303 is added to the first character in the lyrics text corresponding to the verse audio of the sample audio, and a chorus mark 304 is added to the first character in the lyrics text corresponding to the chorus audio of the sample audio.

[0043] At least one positive audio sample constitutes a positive audio sample set, and at least one negative audio sample constitutes a negative audio sample set. Positive audio sample refers to audio that meets audio generation conditions, while negative audio sample refers to audio that does not meet the audio generation conditions. The audio generation conditions are conditions related to audio quality. Audio quality includes at least one of the audio quality, the musicality of the audio, and the degree of genre matching of the audio.

[0044] In some embodiments, the audio generation condition includes at least one of the following: the audio's sound quality score satisfies a condition, the audio's musicality score satisfies a condition, or the audio's genre matching score satisfies a condition. The sound quality score quantifies the audio's sound quality; the musicality score quantifies the audio's musicality; and the genre matching score quantifies the degree of match between the audio and the audio's genre description text.

[0045] Optionally, the positive sample audio is audio that meets an audio generation condition, and the negative sample audio is audio that does not meet the audio generation condition. For example, the positive sample audio is audio whose sound quality score meets the condition, or the positive sample audio is audio whose musicality score meets the condition, or the positive sample audio is audio whose genre matching score meets the condition.

[0046] Optionally, the positive sample audio is audio that meets multiple audio generation conditions, and the negative sample audio is audio that does not meet any of the multiple audio generation conditions. If the positive sample is audio that meets two audio generation conditions, then the negative sample audio is audio that does not meet any of the two audio generation conditions. For example, if the positive sample audio is audio whose sound quality score meets the conditions, and whose musicality score meets the conditions, then the negative sample audio is audio whose sound quality score does not meet the conditions, or the negative sample audio is audio whose musicality score does not meet the conditions. If the positive sample is audio that meets three audio generation conditions, then the negative sample audio is audio that does not meet any of the three audio generation conditions. For example, if the positive sample audio is audio whose sound quality score meets the conditions, and whose musicality score meets the conditions, and whose genre matching score meets the conditions, then the negative sample audio is audio whose sound quality score does not meet the conditions, or the negative sample audio is audio whose musicality score does not meet the conditions, or the negative sample audio is audio whose genre matching score does not meet the conditions.

[0047] In step 220 , the audio generation model is used to obtain a first logarithmic probability corresponding to the positive sample audio and a second logarithmic probability corresponding to the negative sample audio based on the first training sample.

[0048] The first logarithmic probability corresponding to the positive sample audio is used to characterize the preference degree of the first prompt text predicted by the audio generation model for the positive sample audio, and the second logarithmic probability corresponding to the negative sample audio is used to characterize the preference degree of the first prompt text predicted by the audio generation model for the negative sample audio.

[0049] In some embodiments, a first splicing sequence is obtained based on a first prompt text and at least one positive sample audio, and a second splicing sequence is obtained based on the first prompt text and at least one negative sample audio; a first logarithmic probability corresponding to the positive sample audio is generated based on the first splicing sequence through an audio generation model; and a second logarithmic probability corresponding to the negative sample audio is generated based on the second splicing sequence through an audio generation model.

[0050] The first prompt text and at least one positive audio sample are concatenated to obtain a first concatenated sequence. The first concatenated sequence is input into an audio generation model, and the audio generation model outputs a first logarithmic probability corresponding to the positive audio sample. Optionally, the first prompt text and one positive audio sample can be concatenated to obtain the first concatenated sequence, or the first prompt text and multiple positive audio samples can be concatenated to obtain the first concatenated sequence. For example, the first prompt text and all positive audio samples can be concatenated to obtain the first concatenated sequence.

[0051] The audio generation model predicts the probability distribution of the first prompt text for the at least one positive audio sample, and takes the logarithm of the probability distribution of the first prompt text for the at least one positive audio sample to obtain a first logarithmic probability corresponding to the positive audio sample. The first logarithmic probability corresponding to the positive audio sample refers to the logarithmic probability of the first prompt text predicted by the audio generation model in response to the at least one positive audio sample, and is used to indicate the degree of preference of the first prompt text predicted by the audio generation model for the at least one positive audio sample.

[0052] The first prompt text and at least one negative audio sample are concatenated to obtain a second concatenated sequence. The second concatenated sequence is input into the audio generation model, and the audio generation model outputs a second logarithmic probability corresponding to the negative audio sample. Optionally, the first prompt text and one negative audio sample can be concatenated to obtain the second concatenated sequence, or the first prompt text and multiple negative audio samples can be concatenated to obtain the second concatenated sequence. For example, the first prompt text and all negative audio samples can be concatenated to obtain the second concatenated sequence.

[0053] The audio generation model predicts the probability distribution of the first prompt text for the at least one negative audio sample, takes the logarithm of the probability distribution of the first prompt text for the at least one negative audio sample, and obtains a second logarithmic probability corresponding to the negative audio sample. The second logarithmic probability corresponding to the negative audio sample is the logarithmic probability of the response of the first prompt text to the at least one negative audio sample predicted by the audio generation model, and is used to indicate the degree of preference of the first prompt text predicted by the audio generation model for the at least one negative audio sample.

[0054] By using an audio generation model to predict the preference of the first prompt text for positive sample audio and negative sample audio, the audio generation model can be subsequently trained based on the first logarithmic probability and the second logarithmic probability to have the ability to understand and predict text preferences, so that the audio generated by the trained audio generation model can meet the preferences corresponding to the positive sample audio, thereby improving the audio generation quality of the trained audio generation model.

[0055] Step 230 : Obtain a first reference logarithmic probability corresponding to the positive sample audio and a second reference logarithmic probability corresponding to the negative sample audio according to the first training sample using a reference strategy model.

[0056] The first reference logarithmic probability corresponding to the positive sample audio is used to indicate the degree of preference of the first prompt text predicted by the current strategy model for the positive sample audio, and the second reference logarithmic probability corresponding to the negative sample audio is used to indicate the degree of preference of the first prompt text predicted by the current strategy model for the negative sample audio. The current strategy model is the model obtained during the training process of the audio generation model.

[0057] In some embodiments, a first reference logarithmic probability corresponding to the positive sample audio is generated according to the first splicing sequence using a reference strategy model; and a second reference logarithmic probability corresponding to the negative sample audio is generated according to the second splicing sequence using a reference strategy model.

[0058] Optionally, the reference strategy model may be an audio generation model corresponding to an intermediate state in the audio generation model training process, and may be an audio generation model obtained after at least one previous sample training. The audio generation model obtained after the current sample training will serve as the reference strategy model for the next sample training, and training will continue until the final sample training is completed. The resulting audio generation model is the trained audio generation model.

[0059] Optionally, the reference strategy model may be a model trained by other training methods. The structure of the reference model may be the same as or different from the structure of the audio generation model. For example, the structure of the reference strategy model is more complex than that of the audio generation model, or the training method of the reference strategy model is better.

[0060] The first spliced sequence is input into the reference strategy model, and the reference strategy model outputs the first reference logarithmic probability corresponding to the positive sample audio. The reference strategy model predicts the probability distribution of the first prompt text for at least one positive sample audio, takes the logarithm of the probability distribution of the first prompt text for at least one positive sample audio, and obtains the first reference logarithmic probability corresponding to the positive sample audio. The first reference logarithmic probability corresponding to the positive sample audio refers to the logarithmic probability of the first prompt text predicted by the reference strategy model in response to at least one positive sample audio, and is used to characterize the preference degree of the first prompt text predicted by the reference strategy model for at least one positive sample audio.

[0061] The second concatenated sequence is input into the reference strategy model, which outputs a second reference logarithmic probability corresponding to the negative audio sample. The reference strategy model predicts the probability distribution of the first prompt text for at least one negative audio sample, takes the logarithm of the probability distribution of the first prompt text for at least one negative audio sample, and obtains the second reference logarithmic probability corresponding to the negative audio sample. The second reference logarithmic probability corresponding to the negative audio sample refers to the logarithmic probability of the first prompt text in response to at least one negative audio sample predicted by the reference strategy model, and is used to characterize the degree of preference of the first prompt text predicted by the reference strategy model for at least one negative audio sample.

[0062] Step 240: Train the audio generation model based on the first logarithmic probability, the first reference logarithmic probability, the second logarithmic probability, and the second reference logarithmic probability to obtain a trained audio generation model.

[0063] In some embodiments, a first logarithmic probability difference is obtained based on the first logarithmic probability and the second logarithmic probability; a second logarithmic probability difference is obtained based on the first reference logarithmic probability and the second reference logarithmic probability; a first loss function value is calculated based on the first logarithmic probability difference and the second logarithmic probability difference; and based on the first loss function value, the parameters of the audio generation model are adjusted to obtain a trained audio generation model.

[0064] The first logarithmic probability corresponding to the positive sample audio is subtracted from the second logarithmic probability corresponding to the negative sample audio to obtain a first logarithmic probability difference. The first logarithmic probability difference is used to indicate the tendency of the audio generation model for preferential response, that is, to indicate the tendency of the audio generation model to respond to positive sample audio and negative sample audio.

[0065] The first reference logarithmic probability corresponding to the positive sample audio is subtracted from the second reference logarithmic probability corresponding to the positive sample audio to obtain a second logarithmic probability difference. The second logarithmic probability difference is used to indicate the tendency of the reference strategy model for preferential response, that is, to indicate the tendency of the reference strategy model to respond to positive sample audio and negative sample audio.

[0066] Exemplarily, the first loss function value can be expressed as:

[0067] DPO Loss=-log sigmoid(β·(p pi -p ref ))

[0068] Among them, β represents the temperature parameter, p pi represents the second log-probability difference, p ref represents the first log-probability difference.

[0069] With the goal of reducing the first loss function, the parameters of the reference strategy model are adjusted to obtain a trained audio generation model. This trained audio generation model is obtained through multiple rounds of iterative updates.

[0070] By executing the above-mentioned preference optimization training steps, the audio generated by the trained audio generation model can be biased towards the audio features of the positive sample audio. When the positive sample audio meets the audio generation conditions, the preference optimization training can encourage the trained audio generation model to generate audio that meets the audio generation conditions. For example, if the audio generation condition means that the sound quality score of the audio meets the condition, then the preference optimization training is used to encourage the trained audio generation model to generate audio whose sound quality score meets the condition. This enables the trained audio generation model to have the ability to understand and predict text preferences, thereby improving the audio generation quality of the trained audio generation model.

[0071] The technical solution provided in the embodiment of the present application, by using an audio generation model and a reference strategy model to respectively predict the preference of the first prompt text for positive sample audio and negative sample audio, can obtain the tendencies of the audio generation model and the reference strategy model to respond to positive sample audio and negative sample audio respectively, so that after performing preference optimization training on the audio generation model, the audio generated by the trained audio generation model can be biased towards the audio features of the positive sample audio, so that the trained audio generation model can generate audio that meets the audio generation conditions. Compared with the related art of only performing audio generation training on the audio generation model, the technical solution of the present application allows the trained audio generation model to learn the ability to understand and predict preferences, more efficiently capture the subtle differences in preferences, so that the generated audio can meet the preferences corresponding to the positive sample audio, so that the quality of the generated audio can meet the audio generation conditions, and fully improve the audio generation quality of the audio generation model.

[0072] Next, a method for obtaining at least one positive sample audio (hereinafter referred to as a positive sample audio set) and at least one negative sample audio (hereinafter referred to as a negative sample audio set) in the first training sample is introduced.

[0073] In some embodiments, the audio generation condition includes satisfying a sound quality score of the audio. At least two candidate audio samples in the candidate data set are respectively input into a sound quality analysis model, which outputs sound quality scores corresponding to the at least two candidate audio samples. Candidate audio samples with sound quality scores greater than or equal to a first threshold are determined as positive audio samples; and candidate audio samples with sound quality scores less than the first threshold are determined as negative audio samples.

[0074] The sound quality score is used to quantify the audio quality. Audio quality includes at least one of the following dimensions: audio clarity, audio bit depth, audio balance, audio stereo, audio distortion, and audio signal-to-noise ratio. The sound quality score quantifies the audio quality of at least one dimension. The audio quality score meeting the condition means that the audio quality score is greater than or equal to the first threshold.

[0075] The candidate dataset is a pre-set dataset used to generate a positive sample audio set and a negative sample audio set. At least two candidate sample audios in the candidate dataset are obtained and respectively input into a sound quality analysis model. The sound quality analysis model outputs sound quality scores corresponding to the at least two candidate sample audios. Based on the sound quality scores corresponding to the at least two candidate sample audios, the at least two candidate sample audios in the candidate dataset are classified into a positive sample audio set and a negative sample audio set.

[0076] The sound quality analysis model is a pre-trained model for outputting the sound quality score of audio. The sound quality analysis model uses the sound quality differences of the candidate sample audio (such as clarity, bit depth, balance, stereo, distortion, signal-to-noise ratio, etc.) to output the sound quality scores of each candidate sample audio, and then distinguishes between positive sample audio and negative sample audio based on the sound quality scores of each candidate sample audio. Candidate sample audio with a sound quality score greater than or equal to the first threshold is determined as positive sample audio, and candidate sample audio with a sound quality score less than the first threshold is determined as negative sample audio. The first threshold is set by the technical staff according to the sound quality conditions of the positive sample audio, and this application does not limit it.

[0077] Optionally, the sound quality analysis model can also be a pre-trained two-classification model based on sound quality scores, and the sound quality analysis model uses the sound quality differences (such as clarity, bit depth, balance, stereo, distortion, signal-to-noise ratio, etc.) of the candidate sample audio to distinguish the audio type of the candidate sample audio. At least two candidate sample audios in the candidate data set are respectively input into the sound quality analysis model, and the sound quality analysis model outputs the sound quality analysis results corresponding to at least two candidate sample audios. When the sound quality analysis result of the candidate sample audio is a first value, the candidate sample audio is determined to be a positive sample audio, and when the sound quality analysis result of the candidate sample audio is a second value, the candidate sample audio is determined to be a negative sample audio. Exemplarily, the first value can be 1 and the second value can be 0. In the case where the sound quality analysis result of the candidate sample audio is 1, the candidate sample audio is determined to be a positive sample audio, and when the sound quality analysis result of the candidate sample audio is 0, the candidate sample audio is determined to be a negative sample audio.

[0078] By distinguishing positive sample audio and negative sample audio based on the audio quality score, after performing preference optimization training on the trained audio generation model, the re-trained audio generation model can be prompted to generate audio with a sound quality score that meets the conditions, thereby improving the audio generation quality at the sound quality level.

[0079] In some embodiments, the audio generation condition includes the audio's musicality score satisfying a condition. At least two candidate audio samples in the candidate data set are respectively input into a musicality analysis model, which outputs musicality scores corresponding to the at least two candidate audio samples. Candidate audio samples with musicality scores greater than or equal to a second threshold are determined as positive audio samples; and candidate audio samples with musicality scores less than the second threshold are determined as negative audio samples.

[0080] The musicality score is used to quantitatively represent the musicality of an audio track. The musicality of an audio track includes at least one of the following dimensions: the rationality of the audio's compositional structure, the rationality of its musical theory, and the rationality of its chordal rules. The musicality score quantitatively represents the musicality of the audio track in at least one dimension. The condition that the musicality score of an audio track satisfies the condition means that the musicality score of the audio track is greater than or equal to the second threshold.

[0081] The candidate dataset is a pre-set dataset used to generate a positive sample audio set and a negative sample audio set. At least two candidate sample audios in the candidate dataset are obtained and respectively input into a musicality analysis model. The musicality analysis model outputs musicality scores corresponding to the at least two candidate sample audios. Based on the musicality scores corresponding to the at least two candidate sample audios, the at least two candidate sample audios in the candidate dataset are classified into a positive sample audio set and a negative sample audio set.

[0082] The musicality analysis model is a pre-trained model for outputting the musicality score of audio. The musicality analysis model uses the musicality differences of the candidate sample audio (such as composition structure, music theory rules, chord rules, etc.) to output the musicality scores of each candidate sample audio, and then distinguishes between positive sample audio and negative sample audio based on the musicality scores of each candidate sample audio. Candidate sample audio with a musicality score greater than or equal to the second threshold is determined as positive sample audio, and candidate sample audio with a musicality score less than the second threshold is determined as negative sample audio. The second threshold is set by the technician according to the musicality conditions of the positive sample audio, and this application does not limit it.

[0083] Optionally, the musicality analysis model can also be a pre-trained two-classification model based on musicality scores, and the musicality analysis model uses the musicality differences of the candidate sample audio (such as composition structure, music theory rules, chord rules, etc.) to distinguish the audio types of the candidate sample audio. At least two candidate sample audios in the candidate data set are respectively input into the musicality analysis model, and the musicality analysis model outputs the musicality analysis results corresponding to at least two candidate sample audios. When the musicality analysis result of the candidate sample audio is a third numerical value, the candidate sample audio is determined as a positive sample audio, and when the audio analysis result of the candidate sample audio is a fourth numerical value, the candidate sample audio is determined as a negative sample audio. Exemplarily, the third numerical value can be 1 and the fourth numerical value can be 0. In the case where the musicality analysis result of the candidate sample audio is 1, the candidate sample audio is determined as a positive sample audio, and when the musicality analysis result of the candidate sample audio is 0, the candidate sample audio is determined as a negative sample audio.

[0084] By distinguishing positive sample audio from negative sample audio based on the musicality score of the audio, after performing preference optimization training on the trained audio generation model, the re-trained audio generation model can be prompted to generate audio with a musicality score that meets the conditions, thereby improving the generation quality of the audio at the musicality level.

[0085] In some embodiments, the musicality analysis model includes: a structure detection model, a music theory analysis model, and a chord analysis model. At least two candidate sample audios are respectively input into the structure detection model, and the structure detection model outputs the structure scores corresponding to the at least two candidate sample audios. The structure scores are used to quantify the rationality of the composition structure of the candidate sample audios. At least two candidate sample audios are respectively input into the music theory analysis model, and the music theory analysis model outputs the music theory scores corresponding to the at least two candidate sample audios. The music theory scores are used to quantify the rationality of the music theory rules of the candidate sample audios. At least two candidate sample audios are respectively input into the chord analysis model, and the chord analysis model outputs the chord scores corresponding to the at least two candidate sample audios. The chord scores are used to quantify the rationality of the chord rules of the candidate sample audios. For each of the at least two candidate sample audios, the musicality score corresponding to the candidate sample audio is obtained based on the structure score, music theory score, and chord score corresponding to the candidate sample audio.

[0086] The structure detection model is a pre-trained model that outputs a structure score for audio. It uses the matching and clarity between the various structural components of a candidate audio sample to output a corresponding structure score for each candidate audio sample. The structure score for a candidate audio sample quantifies the rationality of the candidate audio sample's structural composition, and can also be understood as quantifying the matching and clarity between the various structural components of the candidate audio sample. The music theory analysis model is a pre-trained model that outputs a music theory score for audio. It uses the music theory rules in the candidate audio sample to output a music theory score for each candidate audio sample. The music theory score for a candidate audio sample quantifies the rationality of the music theory rules in the candidate audio sample, and can also be understood as the correctness of the music theory methods in the candidate audio sample. The chord analysis model is a pre-trained model that outputs a chord score for audio. It uses the chord rules in the candidate audio sample to output a corresponding chord score for each candidate audio sample. The chord score for a candidate audio sample quantifies the rationality of the chord rules in the candidate audio sample, and can also be understood as the correctness of the chord methods in the candidate audio sample.

[0087] For each of the at least two candidate sample audios, the structure score, music theory score, and chord score corresponding to the candidate sample audio are weighted to obtain a musicality score corresponding to the candidate sample audio. The weight of the structure score corresponding to the candidate sample audio is a first weight, the weight of the music theory score corresponding to the candidate sample audio is a second weight, and the weight of the chord score corresponding to the candidate sample audio is a third weight. Then, the musicality score corresponding to the candidate sample audio = the structure score corresponding to the candidate sample audio × the first weight + the music theory score corresponding to the candidate sample audio × the second weight + the chord score corresponding to the candidate sample audio × the third weight.

[0088] The specific values of the first weight, the second weight, and the third weight are set by the technician according to the musicality conditions of the positive sample audio, and this application does not limit them. Optionally, the first weight, the second weight, and the third weight can be the same, or there can be one weight that is different from the other weights. Exemplarily, the first weight, the second weight, and the third weight are the same, all 1, then the musicality score corresponding to the candidate sample audio = the structural score corresponding to the candidate sample audio + the music theory score corresponding to the candidate sample audio + the chord score corresponding to the candidate sample audio.

[0089] By using a structure detection model, a music theory analysis model, and a chord analysis model to analyze and score the candidate sample audio on multiple dimensions of musicality, the final musicality score can comprehensively consider multiple musicality dimensions of the audio, thereby improving the accuracy of the musicality score, thereby improving the accuracy of the classification of positive and negative sample audio, and further helping to improve the accuracy of the model's execution preference optimization training.

[0090] In some embodiments, the audio generation condition includes satisfying a genre matching score of the audio. At least two candidate audio samples in the candidate data set and the genre description texts corresponding to the at least two candidate audio samples are respectively input into a genre matching model, which then outputs genre matching scores corresponding to the at least two candidate audio samples. Candidate audio samples with genre matching scores greater than or equal to a third threshold are determined as positive sample audio; and candidate audio samples with genre matching scores less than the third threshold are determined as negative sample audio.

[0091] The genre matching score is used to quantify the degree of matching between the audio and the genre description text of the audio. The genre matching score of the audio meets the condition, which means that the genre matching score of the audio is greater than or equal to the third threshold.

[0092] The candidate dataset is a pre-set dataset used to generate a positive sample audio set and a negative sample audio set. At least two candidate sample audios in the candidate dataset are obtained and respectively input into a genre matching model. The genre matching model then outputs genre matching scores corresponding to the at least two candidate sample audios. Based on the genre matching scores corresponding to the at least two candidate sample audios, the at least two candidate sample audios in the candidate dataset are classified into a positive sample audio set and a negative sample audio set.

[0093] The style matching model is a pre-trained model for outputting the style matching score of audio. The style matching model uses the matching degree between the candidate sample audio and the style description text of the candidate sample audio to output the style matching score of each candidate sample audio, and then distinguishes between positive sample audio and negative sample audio based on the style matching score of each candidate sample audio. Candidate sample audio with a style matching score greater than or equal to the third threshold is determined as positive sample audio, and candidate sample audio with a style matching score less than the third threshold is determined as negative sample audio. The third threshold is set by the technician according to the style matching condition of the positive sample audio, and this application does not limit it.

[0094] Optionally, the style matching model can also be a pre-trained binary classification model based on the style matching score, and the style matching model uses the matching degree between the candidate sample audio and the style description text of the candidate sample audio to distinguish the audio type of the candidate sample audio. At least two candidate sample audios in the candidate data set are respectively input into the style matching model, and the style matching model outputs the style matching results corresponding to the at least two candidate sample audios. When the style matching result of the candidate sample audio is the fifth numerical value, the candidate sample audio is determined as a positive sample audio, and when the style matching result of the candidate sample audio is the sixth numerical value, the candidate sample audio is determined as a negative sample audio. Exemplarily, the fifth numerical value can be 1 and the sixth numerical value can be 0. When the style matching result of the candidate sample audio is 1, the candidate sample audio is determined as a positive sample audio, and when the style matching result of the candidate sample audio is 0, the candidate sample audio is determined as a negative sample audio.

[0095] By distinguishing positive sample audio and negative sample audio based on the audio genre matching score, after performing preference optimization training on the trained audio generation model, the re-trained audio generation model can be prompted to generate audio with a genre matching score that meets the conditions, thereby improving the audio generation quality at the genre matching level when the audio description text includes genre description text.

[0096] In some embodiments, the audio generation condition includes that the audio's sound quality score satisfies a condition, and the audio's musicality score satisfies a condition. At least two candidate audio samples in the candidate data set are respectively input into a sound quality analysis model, which outputs the sound quality scores corresponding to the at least two candidate audio samples. Furthermore, at least two candidate audio samples in the candidate data set are respectively input into a musicality analysis model, which outputs the musicality scores corresponding to the at least two candidate audio samples.

[0097] Candidate sample audio having a sound quality score greater than or equal to a first threshold and a musicality score greater than or equal to a second threshold is determined as a positive sample audio, and other candidate sample audios are determined as negative sample audios. The other candidate sample audios here include candidate sample audio having a sound quality score greater than or equal to the first threshold and a musicality score less than the second threshold, candidate sample audio having a sound quality score less than the first threshold and a musicality score greater than or equal to the second threshold, and candidate sample audio having a sound quality score less than the first threshold and a musicality score less than the second threshold.

[0098] In some embodiments, the audio generation condition includes that the audio's sound quality score satisfies a condition, and the audio's genre matching score satisfies a condition. Then, at least two candidate audio samples in the candidate data set are respectively input into a sound quality analysis model, and the sound quality analysis model outputs the sound quality scores corresponding to the at least two candidate audio samples, and the at least two candidate audio samples in the candidate data set and the genre description texts corresponding to the at least two candidate audio samples are respectively input into a genre matching model, and the genre matching model outputs the genre matching scores corresponding to the at least two candidate audio samples.

[0099] The candidate sample audio having a sound quality score greater than or equal to the first threshold and a music style matching score greater than or equal to the third threshold is determined as a positive sample audio, and the other candidate sample audios are determined as negative sample audios. The other candidate sample audios here include candidate sample audio having a sound quality score greater than or equal to the first threshold and a music style matching score less than the third threshold, candidate sample audio having a sound quality score less than the first threshold and a music style matching score greater than or equal to the third threshold, and candidate sample audio having a sound quality score less than the first threshold and a music style matching score less than the third threshold.

[0100] In some embodiments, the audio generation condition includes the audio's musicality score satisfying a condition and the audio's genre matching score satisfying a condition. At least two candidate audio samples in the candidate data set are respectively input into a musicality analysis model, which outputs musicality scores corresponding to the at least two candidate audio samples. Furthermore, at least two candidate audio samples in the candidate data set and genre description texts corresponding to the at least two candidate audio samples are respectively input into a genre matching model, which outputs genre matching scores corresponding to the at least two candidate audio samples.

[0101] The candidate sample audio whose musicality score is greater than or equal to the second threshold and whose style matching score is greater than or equal to the third threshold is determined as a positive sample audio, and the other candidate sample audios are determined as negative sample audios. The other candidate sample audios here include candidate sample audio whose musicality score is greater than or equal to the second threshold and whose style matching score is less than the third threshold, candidate sample audio whose musicality score is less than the second threshold and whose style matching score is greater than or equal to the third threshold, and candidate sample audio whose musicality score is less than the second threshold and whose style matching score is less than the third threshold.

[0102] In some embodiments, the above method also includes at least one step from steps 250 to 280, where steps 250 to 280 are a pre-training process of the audio generation model, which is used to train the initial audio generation model to obtain the audio generation model used in the above step 220.

[0103] Step 250: Acquire a second training data set, where the second training data set includes at least one second training sample, and each second training sample includes a second prompt text and a second sample audio.

[0104] The second prompt text comprises the audio description text of the second sample audio and the lyrics text of the second sample audio. For the introduction of the second prompt text, reference can be made to the above-mentioned introduction about the first prompt text, and this application does not go into details. Optionally, the second sample audio can be identical with the first sample audio, or different from the first sample audio. Optionally, when the second sample audio is identical with the first sample audio, the second prompt text can be identical with the first prompt text, or different from the first prompt text.

[0105] Step 260: Obtain predicted audio corresponding to the second prompt text based on the second prompt text and the second sample audio through the initial audio generation model.

[0106] In some embodiments, step 260 includes at least one of sub-steps 261 - 265 .

[0107] Sub-step 261 , performing feature extraction on the second prompt text and the second sample audio respectively to obtain text features of the second prompt text and audio features of the second sample audio.

[0108] Exemplarily, the second prompt text is input into a text encoder, which outputs text features of the second prompt text. The second sample audio is input into an audio encoder, which outputs audio features of the second sample audio.

[0109] Sub-step 262: obtaining audio features of a first proportion from the audio features of the second sample audio.

[0110] Exemplarily, the audio features of the second sample audio are in vector form, and obtaining a first proportion of audio features in the audio features of the second sample audio refers to obtaining a first proportion of tokens in each token of the audio features of the second sample audio.

[0111] The first ratio is set by the technician according to the audio generation requirements and is not limited by this application. For example, if the first ratio is 50%, then step 262 is to obtain 50% of the tokens of the audio features of the second sample audio.

[0112] Sub-step 263 : replacing the second proportion of audio features in the first proportion of audio features with fixed features, and replacing the third proportion of audio features in the first proportion of audio features with random features, to obtain replaced audio features.

[0113] Exemplarily, for the first proportion of tokens that have been obtained, the second proportion of tokens are replaced with fixed features, and the third proportion of tokens are replaced with random features, and the remaining proportion of tokens remain unchanged. The second and third proportions are set by technical personnel according to the audio generation requirements, and this application does not limit them. Exemplarily, the first proportion is 50%, the second proportion is 80%, and the third proportion is 10%. Among the 50% of tokens selected, 80% of the tokens are replaced with fixed features, 10% of the tokens are replaced with random features, and the remaining proportion of tokens remain unchanged. Therefore, the replaced audio features include 55% of unchanged audio features, 40% of fixed features, and 5% of random features.

[0114] A fixed feature refers to the same feature, for example, a fixed feature can be a [mask] tag. A random feature refers to a randomly selected feature, for example, a random feature can be any number.

[0115] In sub-step 264 , the text features of the second prompt text and the replaced audio features are input into the initial audio generation model, and the initial audio generation model outputs the audio features of the predicted audio corresponding to the second prompt text.

[0116] The text features of the second prompt text and the replaced audio features are spliced to obtain a spliced feature sequence, and the spliced feature sequence is input into the initial audio generation model, which outputs the audio features of the predicted audio corresponding to the second prompt text.

[0117] In sub-step 265 , the audio features of the predicted audio are input into the decoding and restoration model, and the decoding and restoration model outputs the predicted audio.

[0118] The decoding and restoration model is used to convert the feature representation back to its original form, such as mel-spectrogram or audio. Therefore, the audio features of the predicted audio are input into the decoding and restoration model, which then outputs the predicted audio. For example, the predicted audio can be in the form of mel-spectrogram.

[0119] By masking the audio features of the second sample audio, the initial audio generation model can restore the masked audio features based on the text features of the second prompt text, thereby improving the detail understanding and analysis capabilities of the audio generation model, thereby helping to improve the audio generation quality of the audio generation model.

[0120] Step 270: Calculate a second loss function value based on the difference between the predicted audio and the second sample audio.

[0121] Exemplarily, the second loss function value is calculated based on the difference between the audio features of the predicted audio and the audio features of the second sample audio.

[0122] After executing sub-step 264 , sub-step 270 may be executed directly without executing sub-step 265 , thereby avoiding re-acquisition of audio features of the predicted audio.

[0123] For the second loss function value, a mean square error loss function may be used. For example, the second loss function value may be expressed as:

[0124]

[0125] Where n represents the number of audio features, y i represents the i-th audio feature of the second sample audio, Represents the i-th audio feature of the predicted audio.

[0126] Step 280: Adjust the parameters of the initial audio generation model according to the second loss function value to obtain the audio generation model.

[0127] With the goal of reducing the second loss function, the parameters of the initial audio generation model are adjusted to obtain the audio generation model. This audio generation model needs to be obtained through multiple rounds of iterative updates.

[0128] By performing the pre-training steps of the above audio generation training, the generated audio can match the prompt text, thereby improving the audio generation quality of the audio generation model.

[0129] Please refer to Figure 4 , which shows a flow chart of a complete training method for an audio generation model provided by one embodiment of the present application. The execution subject of each step of the method can be a computer device. The method may include at least one of the following steps 410 to 430:

[0130] Step 410: Obtain a first training data set and a second training data set of an initial audio generation model, where the first training data set includes at least one first training sample, and the second training data set includes at least one second training sample. Each first training sample includes a first prompt text, at least one positive sample audio, and at least one negative sample audio, and each second training sample includes a second prompt text and a second sample audio.

[0131] The first prompt text includes the audio description text and lyrics text of the first audio sample, and the second prompt text includes the audio description text and lyrics text of the second audio sample. Positive audio samples are audio that meets the audio generation conditions, while negative audio samples are audio that does not meet the audio generation conditions. The audio generation conditions are conditions related to the audio quality.

[0132] Step 420: Perform a pre-training process on the initial audio generation model based on the second training data set to obtain an audio generation model. The pre-training process is used to enable the audio generation model to generate audio that matches the second prompt text.

[0133] The specific process of step 420 can refer to the execution process of the above steps 250 to 280, which will not be described in detail here.

[0134] Step 430: Perform preference optimization training on the audio generation model based on the first training data set to obtain a trained audio generation model. The preference optimization training is used to encourage the trained audio generation model to generate audio that meets the audio generation conditions.

[0135] The specific process of step 430 can refer to the execution process of the above steps 210 to 240, which will not be described in detail here.

[0136] Please refer to Figure 5 , which shows a flow chart of an audio generation method based on an audio generation model provided by one embodiment of the present application. The execution subject of each step of the method can be a computer device. The method can include at least one of the following steps 510 to 530:

[0137] Step 510: Obtain input prompt text, where the input prompt text includes the audio description text of the audio to be generated and the lyrics text of the audio to be generated.

[0138] The audio description text of the audio to be generated is used to describe the audio features of the audio to be generated. For details, please refer to the above embodiment and will not be described in detail here.

[0139] In step 520 , the trained audio generation model is used to obtain output audio features corresponding to the input prompt text according to the input prompt text. The output audio features are audio features of the audio to be generated.

[0140] In some embodiments, step 520 includes at least one of sub-steps 521 - 524 .

[0141] Sub-step 521 , performing feature extraction on the input prompt text to obtain text features of the input prompt text.

[0142] Exemplarily, the input prompt text is input into a text encoder, and the text encoder outputs text features of the input prompt text.

[0143] For example, the audio description text and the lyrics text of the audio to be generated are respectively input into a text encoder, and the text encoder outputs the text features of the audio description text and the lyrics text, respectively. The text features of the input prompt text are obtained based on the text features of the output audio description text and the text features of the lyrics text.

[0144] In sub-step 522, the text features of the input prompt text are input into the trained audio generation model, and the trained audio generation model outputs the first output audio features corresponding to the input prompt text.

[0145] In the first regression process, the text features of the input prompt text and the first output audio features are input into the trained audio generation model, and the trained audio generation model outputs the second output audio features corresponding to the input prompt text.

[0146] In sub-step 523, during the i-th regression process, the text features of the input prompt text and the previous i-time output audio features are input into the trained audio generation model, and the trained audio generation model outputs the i+1-th output audio features corresponding to the input prompt text, where i is a positive integer.

[0147] The first i output audio features include the first output audio feature, ..., the i-th output audio feature.

[0148] Exemplarily, in the second regression process, the text features of the input prompt text, the first output audio features and the second output audio features are input into the trained audio generation model, and the trained audio generation model outputs the third output audio features corresponding to the input prompt text.

[0149] Sub-step 524 , obtaining output audio features corresponding to the input prompt text according to a preset number of regressions, where the output audio features include at least one output audio feature corresponding to the input prompt text.

[0150] For example, if the number of regressions is preset to N, then during the N-time regression process, the N+1th output audio feature corresponding to the input prompt text is obtained. The N+1th output audio feature corresponding to the input prompt text is the output audio feature corresponding to the input prompt text. The N+1th output audio feature corresponding to the input prompt text includes the 1st output audio feature, ..., the N+1th output audio feature.

[0151] Step 530: Input the output audio features into a decoding and restoration model, and the decoding and restoration model outputs the output audio corresponding to the input prompt text.

[0152] Exemplarily, the N+1 output audio features corresponding to the input prompt text are input into the decoding and restoration model, and the decoding and restoration model outputs the output audio corresponding to the input prompt text.

[0153] By regressing and regenerating the output audio features that have been generated, the accuracy of the output audio features generated next time is improved, so that after multiple regression operations, more accurate output audio features can be obtained, thereby improving the matching degree between the output audio and the input prompt text, and improving the bias of the output audio corresponding to the input prompt text in responding to the positive sample audio, thereby comprehensively improving the generation quality of the output audio.

[0154] Figure 6 This diagram shows the output audio generation process. The input prompt text includes audio description text and lyrics text. Feature extraction is performed on the audio description text and lyrics text respectively to obtain text features of the audio description text and lyrics text. These text features of the audio description text and lyrics text are input into the audio generation model, which outputs the corresponding output audio features. Through multiple regression processes, multiple output audio features are obtained, and the decoding and restoration model outputs the output audio corresponding to the input prompt text.

[0155] Figure 7A flow chart of multiple recursive generation of output audio is shown. The text features of the audio description text and the text features of the lyrics text are input into the audio generation model, and the audio generation model outputs the first output audio features. The audio generation model here is the above-mentioned audio generation model that has been trained through preference optimization. The text features of the audio description text, the text features of the lyrics text and the first output audio features are input into the audio generation model, and the audio generation model outputs the second output audio features. The text features of the audio description text, the text features of the lyrics text, the first output audio features and the second output audio features are input into the audio generation model, and the audio generation model outputs the third output audio features. The regression steps are executed in this manner until the preset number of regressions is reached, and the regression steps are terminated. Finally, N+1 output audio features are obtained, and the N+1 output audio features are input into the decoding and restoration model, which outputs the corresponding output audio.

[0156] The training method of the audio generation model and the audio generation method based on the audio generation model provided in the embodiment of the present application are corresponding model training processes and usage processes. For details not described in detail on one side, please refer to the introduction and description on the other side.

[0157] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0158] Please refer to Figure 8 , which shows a block diagram of a training device for an audio generation model provided by an embodiment of the present application. The device has the function of implementing the training method of the above-mentioned audio generation model, and the function can be implemented by hardware or by hardware executing corresponding software. The device can be the computer device described above, or it can be set in a computer device. Figure 8 As shown, the device 800 may include: a data acquisition module 810, a first generation module 820, a second generation module 830 and a model training module 840.

[0159] The data acquisition module 810 is used to obtain a first training data set, where the first training data set includes at least one first training sample, each first training sample includes a first prompt text, at least one positive sample audio and at least one negative sample audio, where the positive sample audio is audio that meets the audio generation condition, and the negative sample audio is audio that does not meet the audio generation condition, where the audio generation condition is a condition related to the quality of the audio.

[0160] The first generation module 820 is used to obtain a first logarithmic probability corresponding to the positive sample audio and a second logarithmic probability corresponding to the negative sample audio based on the first training sample through the audio generation model. The first logarithmic probability is used to characterize the preference degree of the first prompt text for the positive sample audio, and the second logarithmic probability is used to characterize the preference degree of the first prompt text for the negative sample audio.

[0161] The second generating module 830 is configured to obtain a first reference logarithmic probability corresponding to the positive sample audio and a second reference logarithmic probability corresponding to the negative sample audio according to the first training sample using a reference strategy model.

[0162] The model training module 840 is used to train the audio generation model according to the first logarithmic probability, the first reference logarithmic probability, the second logarithmic probability, and the second reference logarithmic probability to obtain a trained audio generation model.

[0163] In some embodiments, the audio generation condition includes a sound quality score of the audio meeting a condition, where the sound quality score is used to quantitatively represent the sound quality of the audio; the data acquisition module 810 is used to:

[0164] Inputting at least two candidate sample audios in the candidate data set into a sound quality analysis model respectively, and the sound quality analysis model outputs sound quality scores corresponding to the at least two candidate sample audios respectively;

[0165] Determine the candidate sample audio whose sound quality score is greater than or equal to a first threshold as the positive sample audio;

[0166] The candidate sample audio whose sound quality score is less than the first threshold is determined as the negative sample audio.

[0167] In some embodiments, the audio generation condition includes a condition that a musicality score of the audio satisfies a condition, where the musicality score is used to quantitatively represent the musicality of the audio; and the data acquisition module 810 is used to:

[0168] Inputting at least two candidate sample audios in the candidate data set into a musicality analysis model respectively, and the musicality analysis model outputs musicality scores corresponding to the at least two candidate sample audios respectively;

[0169] Determine the candidate sample audio whose musicality score is greater than or equal to a second threshold as the positive sample audio;

[0170] The candidate sample audio whose musicality score is less than the second threshold is determined as the negative sample audio.

[0171] In some embodiments, the musicality analysis model includes: a structure detection model, a music theory analysis model, and a chord analysis model; the training data acquisition module 810 is used to:

[0172] Inputting the at least two candidate audio samples into the structure detection model respectively, and the structure detection model outputs structure scores corresponding to the at least two candidate audio samples respectively, wherein the structure scores are used to quantitatively represent the rationality of the composition structure of the candidate audio samples;

[0173] Inputting the at least two candidate sample audios into the music theory analysis model respectively, and having the music theory analysis model output music theory scores corresponding to the at least two candidate sample audios respectively, wherein the music theory scores are used to quantify the rationality of the music theory rules of the candidate sample audios;

[0174] Inputting the at least two candidate audio samples into the chord analysis model respectively, and having the chord analysis model output chord scores corresponding to the at least two candidate audio samples respectively, wherein the chord scores are used to quantitatively represent the rationality of the chord rules of the candidate audio samples;

[0175] For each candidate sample audio of the at least two candidate sample audios, a musicality score corresponding to the candidate sample audio is obtained according to the structure score, music theory score, and chord score corresponding to the candidate sample audio.

[0176] In some embodiments, the audio generation condition includes a condition that a genre matching score of the audio satisfies a condition, where the genre matching score is used to quantitatively represent a degree of matching between the audio and a genre description text of the audio; the data acquisition module 810 is used to:

[0177] Inputting at least two candidate sample audios in the candidate data set and the genre description texts corresponding to the at least two candidate sample audios into a genre matching model, and the genre matching model outputs the genre matching scores corresponding to the at least two candidate sample audios;

[0178] Determine the candidate sample audio whose genre matching score is greater than or equal to a third threshold as the positive sample audio;

[0179] The candidate sample audio whose genre matching score is less than the third threshold is determined as the negative sample audio.

[0180] In some embodiments, the first generating module 820 is configured to:

[0181] Obtaining a first spliced sequence according to the first prompt text and at least one positive sample audio, and obtaining a second spliced sequence according to the first prompt text and at least one negative sample audio;

[0182] Generating, by the audio generation model, a first logarithmic probability corresponding to the positive sample audio according to the first splicing sequence;

[0183] Generate a second logarithmic probability corresponding to the negative sample audio according to the second splicing sequence through the audio generation model.

[0184] In some embodiments, the model training module 840 is used to:

[0185] Obtaining a first logarithmic probability difference according to the first logarithmic probability and the second logarithmic probability;

[0186] Obtaining a second logarithmic probability difference according to the first reference logarithmic probability and the second reference logarithmic probability;

[0187] Calculating a first loss function value based on the first logarithmic probability difference and the second logarithmic probability difference;

[0188] According to the first loss function value, the parameters of the audio generation model are adjusted to obtain the trained audio generation model.

[0189] In some embodiments, the apparatus 800 further includes a pre-training module, wherein the pre-training module is configured to:

[0190] Acquire a second training data set, where the second training data set includes at least one second training sample, and each second training sample includes a second prompt text and a second sample audio;

[0191] Obtaining, by an initial audio generation model, predicted audio corresponding to the second prompt text according to the second prompt text and the second sample audio;

[0192] Calculating a second loss function value based on a difference between the predicted audio and the second sample audio;

[0193] According to the second loss function value, the parameters of the initial audio generation model are adjusted to obtain the audio generation model.

[0194] In some embodiments, the pre-training module is used to:

[0195] Performing feature extraction on the second prompt text and the second sample audio respectively to obtain text features of the second prompt text and audio features of the second sample audio;

[0196] Obtaining a first proportion of audio features from the audio features of the second sample audio;

[0197] Replacing a second proportion of the audio features in the first proportion with fixed features, and replacing a third proportion of the audio features in the first proportion with random features, to obtain replaced audio features;

[0198] Inputting the text features of the second prompt text and the replaced audio features into the initial audio generation model, and having the initial audio generation model output the audio features of the predicted audio corresponding to the second prompt text;

[0199] The audio features of the predicted audio are input into a decoding and restoration model, and the decoding and restoration model outputs the predicted audio.

[0200] In some embodiments, the apparatus 800 further includes a model application module, wherein the model application module is configured to:

[0201] Obtaining input prompt text, the input prompt text including audio description text of the audio to be generated and lyrics text of the audio to be generated;

[0202] Obtaining, by the trained audio generation model, output audio features corresponding to the input prompt text according to the input prompt text, wherein the output audio features are audio features of the audio to be generated;

[0203] The output audio features are input into a decoding and restoration model, and the decoding and restoration model outputs the output audio corresponding to the input prompt text.

[0204] In some embodiments, the model application module is used to:

[0205] Performing feature extraction on the input prompt text to obtain text features of the input prompt text;

[0206] Inputting the text features of the input prompt text into the trained audio generation model, and having the trained audio generation model output the first output audio features corresponding to the input prompt text;

[0207] In the i-th regression process, the text features of the input prompt text and the previous i-time output audio features are input into the trained audio generation model, and the trained audio generation model outputs the i+1-th output audio features corresponding to the input prompt text, where i is a positive integer;

[0208] According to a preset number of regressions, an output audio feature corresponding to the input prompt text is obtained, where the output audio feature includes at least one output audio feature corresponding to the input prompt text.

[0209] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0210] Please refer to Figure 9 , which shows a block diagram of a computer device 900 provided in one embodiment of the present application. The computer device 900 can be any electronic device with data computing, processing, and storage functions. The computer device 900 can be used to implement the training method of the audio generation model provided in the above embodiment.

[0211] Typically, the computer device 900 includes a processor 901 and a memory 902 .

[0212] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), or PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0213] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store a computer program, which is configured to be executed by one or more processors to implement the above-mentioned audio generation model training method.

[0214] Those skilled in the art will understand that Figure 9 The structure shown in the figure does not constitute a limitation on the computer device 900, and the computer device 900 may include more or fewer components than shown in the figure, or combine some components, or adopt a different component arrangement.

[0215] In an exemplary embodiment, a computer-readable storage medium is also provided, storing a computer program that, when executed by a processor of a computer device, implements the aforementioned audio generation model training method. Optionally, the computer-readable storage medium may be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, or an optical data storage device.

[0216] In an exemplary embodiment, a computer program product is also provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the above-described method for training an audio generation model.

[0217] It should be noted that this application can display a prompt interface, pop-up window or output voice prompt information before collecting the user's relevant data and during the process of collecting the user's relevant data. The prompt interface, pop-up window or voice prompt information is used to remind the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining the user's relevant data after obtaining the user's confirmation operation on the prompt interface or pop-up window. Otherwise (that is, when the user's confirmation operation on the prompt interface or pop-up window is not obtained), the relevant steps of obtaining the user's relevant data are terminated, that is, the user's relevant data is not obtained. In other words, all user data collected by this application are processed strictly in accordance with the requirements of relevant national laws and regulations. The informed consent or separate consent of the personal information subject is obtained only when the user agrees and authorizes it to collect the data. Subsequent data use and processing are carried out within the scope of authorization of laws and regulations and the personal information subject, and the collection, use and processing of relevant user data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0218] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application do not limit this.

[0219] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A training method for an audio generation model, characterized in that: The method comprises: Obtaining a first training data set, where the first training data set includes at least one first training sample, each first training sample including a first prompt text, at least one positive audio sample, and at least one negative audio sample, where the positive audio sample is audio that meets an audio generation condition, and the negative audio sample is audio that does not meet the audio generation condition, where the audio generation condition is a condition related to audio quality; Obtaining, by the audio generation model and based on the first training sample, a first logarithmic probability corresponding to the positive sample audio and a second logarithmic probability corresponding to the negative sample audio, wherein the first logarithmic probability is used to represent the degree of preference of the first prompt text for the positive sample audio, and the second logarithmic probability is used to represent the degree of preference of the first prompt text for the negative sample audio; Obtaining, based on the first training sample and using a reference strategy model, a first reference logarithmic probability corresponding to the positive sample audio and a second reference logarithmic probability corresponding to the negative sample audio; The audio generation model is trained according to the first logarithmic probability, the first reference logarithmic probability, the second logarithmic probability, and the second reference logarithmic probability to obtain a trained audio generation model.

2. The method according to claim 1, characterized in that The audio generation condition includes that a sound quality score of the audio satisfies a condition, where the sound quality score is used to quantitatively represent the sound quality of the audio; the method further includes: Inputting at least two candidate sample audios in the candidate data set into a sound quality analysis model respectively, and the sound quality analysis model outputs sound quality scores corresponding to the at least two candidate sample audios respectively; Determine the candidate sample audio whose sound quality score is greater than or equal to a first threshold as the positive sample audio; The candidate sample audio whose sound quality score is less than the first threshold is determined as the negative sample audio.

3. The method according to claim 1 or 2, characterized in that The audio generation condition includes that the musicality score of the audio satisfies a condition, where the musicality score is used to quantitatively represent the musicality of the audio; the method further includes: Inputting at least two candidate sample audios in the candidate data set into a musicality analysis model respectively, and the musicality analysis model outputs musicality scores corresponding to the at least two candidate sample audios respectively; Determine the candidate sample audio whose musicality score is greater than or equal to a second threshold as the positive sample audio; The candidate sample audio whose musicality score is less than the second threshold is determined as the negative sample audio.

4. The method according to claim 3, characterized in that The musicality analysis model includes: a structure detection model, a music theory analysis model and a chord analysis model; The step of inputting at least two candidate audio samples in the candidate data set into a musicality analysis model, and outputting musicality scores corresponding to the at least two candidate audio samples by the musicality analysis model, comprises: Inputting the at least two candidate audio samples into the structure detection model respectively, and the structure detection model outputs structure scores corresponding to the at least two candidate audio samples respectively, wherein the structure scores are used to quantitatively represent the rationality of the composition structure of the candidate audio samples; Inputting the at least two candidate sample audios into the music theory analysis model respectively, and having the music theory analysis model output music theory scores corresponding to the at least two candidate sample audios respectively, wherein the music theory scores are used to quantify the rationality of the music theory rules of the candidate sample audios; Inputting the at least two candidate audio samples into the chord analysis model respectively, and having the chord analysis model output chord scores corresponding to the at least two candidate audio samples respectively, wherein the chord scores are used to quantitatively represent the rationality of the chord rules of the candidate audio samples; For each candidate sample audio of the at least two candidate sample audios, a musicality score corresponding to the candidate sample audio is obtained according to the structure score, music theory score, and chord score corresponding to the candidate sample audio.

5. The method according to any one of claims 1 to 4, characterized in that The audio generation condition includes a genre matching score of the audio satisfying a condition, where the genre matching score is used to quantitatively represent a degree of matching between the audio and a genre description text of the audio; the method further includes: Inputting at least two candidate sample audios in the candidate data set and the genre description texts corresponding to the at least two candidate sample audios into a genre matching model, and the genre matching model outputs the genre matching scores corresponding to the at least two candidate sample audios; Determine the candidate sample audio whose genre matching score is greater than or equal to a third threshold as the positive sample audio; The candidate sample audio whose genre matching score is less than the third threshold is determined as the negative sample audio.

6. The method according to any one of claims 1 to 5, characterized in that The obtaining, by the audio generation model according to the first training sample, a first logarithmic probability corresponding to the positive sample audio and a second logarithmic probability corresponding to the negative sample audio, includes: Obtaining a first spliced sequence according to the first prompt text and at least one positive sample audio, and obtaining a second spliced sequence according to the first prompt text and at least one negative sample audio; Generating, by the audio generation model, a first logarithmic probability corresponding to the positive sample audio according to the first splicing sequence; Generate a second logarithmic probability corresponding to the negative sample audio according to the second splicing sequence through the audio generation model.

7. The method according to any one of claims 1 to 6, characterized in that The step of training the audio generation model according to the first logarithmic probability, the first reference logarithmic probability, the second logarithmic probability, and the second reference logarithmic probability to obtain a trained audio generation model includes: Obtaining a first logarithmic probability difference according to the first logarithmic probability and the second logarithmic probability; Obtaining a second logarithmic probability difference according to the first reference logarithmic probability and the second reference logarithmic probability; Calculating a first loss function value based on the first logarithmic probability difference and the second logarithmic probability difference; According to the first loss function value, the parameters of the audio generation model are adjusted to obtain the trained audio generation model.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Acquire a second training data set, where the second training data set includes at least one second training sample, and each second training sample includes a second prompt text and a second sample audio; Obtaining, by an initial audio generation model, predicted audio corresponding to the second prompt text according to the second prompt text and the second sample audio; Calculating a second loss function value based on a difference between the predicted audio and the second sample audio; According to the second loss function value, the parameters of the initial audio generation model are adjusted to obtain the audio generation model.

9. The method according to claim 8, characterized in that The obtaining, by using the initial audio generation model according to the second prompt text and the second sample audio, a predicted audio corresponding to the second prompt text includes: Performing feature extraction on the second prompt text and the second sample audio respectively to obtain text features of the second prompt text and audio features of the second sample audio; Obtaining a first proportion of audio features from the audio features of the second sample audio; Replacing a second proportion of the audio features in the first proportion with fixed features, and replacing a third proportion of the audio features in the first proportion with random features, to obtain replaced audio features; Inputting the text features of the second prompt text and the replaced audio features into the initial audio generation model, and having the initial audio generation model output the audio features of the predicted audio corresponding to the second prompt text; The audio features of the predicted audio are input into a decoding and restoration model, and the decoding and restoration model outputs the predicted audio.

10. The method according to any one of claims 1 to 9, characterized in that The method further comprises: Obtaining input prompt text, the input prompt text including audio description text of the audio to be generated and lyrics text of the audio to be generated; Obtaining, by the trained audio generation model, output audio features corresponding to the input prompt text according to the input prompt text, wherein the output audio features are audio features of the audio to be generated; The output audio features are input into a decoding and restoration model, and the decoding and restoration model outputs the output audio corresponding to the input prompt text.

11. The method according to claim 10, characterized in that The step of obtaining, by the trained audio generation model according to the input prompt text, output audio features corresponding to the input prompt text includes: Performing feature extraction on the input prompt text to obtain text features of the input prompt text; Inputting the text features of the input prompt text into the trained audio generation model, and having the trained audio generation model output the first output audio features corresponding to the input prompt text; In the i-th regression process, the text features of the input prompt text and the previous i-time output audio features are input into the trained audio generation model, and the trained audio generation model outputs the i+1-th output audio features corresponding to the input prompt text, where i is a positive integer; According to a preset number of regressions, an output audio feature corresponding to the input prompt text is obtained, where the output audio feature includes at least one output audio feature corresponding to the input prompt text.

12. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the training method of the audio generation model according to any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the training method of the audio generation model according to any one of claims 1 to 11.

14. A computer program product, characterized in that The computer program product comprises a computer program, which is loaded and executed by a processor to implement the method for training an audio generation model according to any one of claims 1 to 11.

Images