Training method of music generation model, music generation method, equipment and medium

Abstract

The invention provides a training method of a music generation model, a music generation method, equipment and a medium, and relates to the technical field of artificial intelligence. Performing first up-sampling training on the training audio coding information based on a first generator in a vocoder in a music generation model to obtain intermediate training audio data of a first sampling rate, the intermediate training audio data is subjected to first up-sampling training based on a first generator in the vocoder, second up-sampling training is carried out on the intermediate training audio data based on a second generator in the vocoder to obtain training audio data of a second sampling rate, the second sampling rate is larger than the first sampling rate, and generative adversarial training is carried out on the vocoder based on the training audio data to obtain a trained music generation model. By adopting the method and the device, the dependence of model training on high-sampling-rate audio data can be reduced while the music generation model is ensured to output high-fidelity songs, so that the model training cost is controlled.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method for a music generation model, a music generation method, a device, and a medium. Background Technology

[0002] Currently, music generation technology, thanks to continuous breakthroughs in artificial intelligence algorithms, has successfully transitioned from fragmented synthesis to complete song creation. Whether modeling a composite audio file containing mixed accompaniment and vocals, or modeling the accompaniment and vocals separately and then merging them to generate a complete segment, music generation models can output high-fidelity songs. However, in the high-fidelity song generation technology system, even though the acoustic model within the music generation model can be trained using low-sampling-rate data, the vocoder training cannot be compatible with the low-cost training output of the acoustic model. This makes the training of both the acoustic model and the vocoder highly dependent on high-sampling-rate audio data, resulting in a significant increase in data storage costs and computational resource consumption during the overall training process. Summary of the Invention

[0003] The main objective of this application is to propose a training method, music generation method, device, and medium for a music generation model, which aims to ensure that the music generation model outputs high-fidelity songs while reducing the model training's dependence on high-sampling-rate audio data, thereby controlling the model training cost.

[0004] To achieve the above objectives, a first aspect of this application proposes a method for training a music generation model. The music generation model includes an acoustic model and a vocoder, wherein the output of the acoustic model is connected to the input of the vocoder, and the method includes: Obtain the training audio encoding information output by the acoustic model training; The training audio coding information is trained by first upsampling based on the first generator in the vocoder to obtain intermediate training audio data with a first sampling rate; The intermediate training audio data is subjected to a second upsampling training based on the second generator in the vocoder to obtain training audio data with a second sampling rate; the second sampling rate is greater than the first sampling rate. Generative adversarial training is performed on the vocoder based on the training audio data to obtain a trained music generation model.

[0005] In some embodiments, the step of performing a first upsampling training on the training audio coding information based on a first generator in the vocoder to obtain intermediate training audio data at a first sampling rate includes: Based on the first generator in the vocoder, the training audio encoding information is encoded into hidden layer tensor information for feature learning, and the hidden layer tensor information is subjected to first upsampling training to obtain intermediate training audio data with a first sampling rate.

[0006] In some embodiments, the first generator includes a first convolutional module and stacked second convolutional modules; The step of encoding the training audio coding information into hidden layer tensor information for feature learning includes: The training audio coding information is encoded into hidden layer tensor information based on the first convolution module; Feature learning processing is performed on the hidden layer tensor information based on the second convolution module, using multi-scale feature fusion.

[0007] In some embodiments, the step of performing a second upsampling training on the intermediate training audio data based on the second generator in the vocoder to obtain training audio data with a second sampling rate includes: The intermediate training audio data is subjected to layer normalization to obtain latent variable information; The latent variable information is input into the second generator in the vocoder; Based on the second generator, the latent variable information is subjected to a second upsampling training to obtain training audio data with a second sampling rate.

[0008] In some embodiments, the second generator includes a one-dimensional transposed convolutional model and an anti-aliasing multi-cycle combination module; The training audio data includes anti-aliasing training audio data, and the step of performing second upsampling training on the latent variable information based on the second generator to obtain training audio data with a second sampling rate includes: The training audio data with a second sampling rate is obtained by performing a second upsampling training on the latent variable information based on the one-dimensional transposed convolution model, and then the training audio data is upsampled and filtered based on the anti-aliasing multi-period combination module to obtain anti-aliasing training audio data.

[0009] In some embodiments, the generative adversarial training of the vocoder based on the training audio data includes: The training audio data is subjected to loss calculation based on a preset discriminator to obtain a first loss calculation result; the discriminator includes a multi-period discriminator, a multi-scale subband constant transformation discriminator and a first multi-resolution spectrum discriminator, and the first loss calculation result includes the sum of the results obtained by the multi-period discriminator, the multi-scale subband constant transformation discriminator and the first multi-resolution spectrum discriminator respectively performing loss calculation on the training audio data; The parameters of the second generator are updated based on the first loss calculation result.

[0010] In some embodiments, the method further includes: The intermediate training audio data is subjected to loss calculation based on a preset discriminator to obtain a second loss calculation result; the discriminator further includes a second multi-resolution spectrum discriminator, and the second loss calculation result is the result obtained by the second multi-resolution spectrum discriminator performing loss calculation on the intermediate training audio data; The parameters of the first generator are updated based on the second loss calculation result.

[0011] To achieve the above objectives, a second aspect of this application proposes a music generation method, the method comprising: Obtain the audio encoding information output by the acoustic model; the acoustic model is the acoustic model in the trained music generation model; The audio encoded information is subjected to a first upsampling process by the first generator in the vocoder to obtain intermediate audio data with a first sampling rate; the vocoder is the vocoder in the trained music generation model. The intermediate audio data is subjected to a second upsampling process based on the second generator in the vocoder to obtain audio data with a second sampling rate.

[0012] To achieve the above objectives, a third aspect of this application provides a training apparatus for a music generation model, wherein the music generation model includes an acoustic model and a vocoder, and the output of the acoustic model is connected to the input of the vocoder; the training apparatus includes: The first acquisition module is used to acquire the training audio encoding information output by the acoustic model training; The first upsampling training module is used to perform first upsampling training on the training audio encoding information based on the first generator in the vocoder to obtain intermediate training audio data with a first sampling rate. The second upsampling training module is used to perform second upsampling training on the intermediate training audio data based on the second generator in the vocoder to obtain training audio data with a second sampling rate; the second sampling rate is greater than the first sampling rate. The adversarial training module is used to perform generative adversarial training on the vocoder based on the training audio data to obtain a trained music generation model.

[0013] To achieve the above objectives, a fourth aspect of this application provides a music generation apparatus, the music generation apparatus comprising: The second acquisition module is used to acquire the audio encoding information output by the acoustic model; the acoustic model is the acoustic model in the trained music generation model. The first upsampling processing module is used to perform a first upsampling process on the audio encoded information based on the first generator in the vocoder to obtain intermediate audio data with a first sampling rate; the vocoder is the vocoder in the trained music generation model. The second upsampling processing module is used to perform a second upsampling process on the intermediate audio data based on the second generator in the vocoder to obtain audio data with a second sampling rate.

[0014] To achieve the above objectives, a fifth aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the training method of the music generation model described in the first aspect and / or the music generation method described in the second aspect.

[0015] To achieve the above objectives, a sixth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the training method of the music generation model described in the first aspect and / or the music generation method described in the second aspect.

[0016] To achieve the above objectives, a seventh aspect of the present application provides a computer program product, which includes a computer program that, when executed by a processor, implements the training method for the music generation model provided in the first aspect above and / or implements the music generation method described in the second aspect above.

[0017] The music generation model training method, music generation device, electronic device, computer-readable storage medium, and computer program product proposed in this application embodiment obtain training audio encoding information output by the acoustic model training in the music generation model, perform first upsampling training on the training audio encoding information based on the first generator in the vocoder in the music generation model to obtain intermediate training audio data with a first sampling rate, and perform second upsampling training on the intermediate training audio data based on the second generator in the vocoder to obtain training audio data with a second sampling rate, wherein the second sampling rate is greater than the first sampling rate, and perform generative adversarial training on the vocoder based on the training audio data to obtain the trained music generation model.

[0018] Thus, this embodiment utilizes a two-stage model structure within the vocoder. The first stage uses a first generator to perform a first upsampling training on the training audio encoding information output by the acoustic model, obtaining intermediate training audio data at a first sampling rate. This intermediate training audio data is further fed to the second stage, where a second generator performs a second upsampling training on the intermediate training audio data, resulting in training audio data at a second sampling rate, which is larger than the first sampling rate. Adversarial training is then performed based on this training audio data to obtain the trained music generation model. Compared to traditional methods where both the acoustic model and vocoder training rely on high-sampling-rate audio data, this embodiment can train the vocoder using low-sampling-rate training data output by the acoustic model at the vocoder's upstream stage. That is, it is compatible with the low-cost training results of the acoustic model using low-sampling-rate data (easy to obtain, large in volume, and low in cost). By using a multi-task approach with progressive upsampling, different sampling rate losses are trained, thereby steadily increasing the model's sampling rate step by step, ultimately training a vocoder capable of outputting high-fidelity audio. In other words, the embodiments of this application can reduce the dependence of model training on high-sampling-rate audio data while ensuring that the music generation model outputs high-fidelity songs, thereby controlling the model training cost.

[0019] Furthermore, in the embodiments of this application, when generating music, the vocoder in the trained music generation model is used to upsample the audio encoding information output by the acoustic model step by step, thereby enabling the direct output of high-fidelity audio data with a high sampling rate based on low sampling rate audio encoding information. Attached Figure Description

[0020] Figure 1 A flowchart illustrating the steps of the training method for the music generation model provided in this application embodiment in some embodiments; Figure 2 for Figure 1 A detailed flowchart of step S102; Figure 3 A schematic diagram of the structure of the first generator Vocos involved in some embodiments of the training method for the music generation model provided in this application. Figure 4 for Figure 1 A detailed flowchart of step S103; Figure 5 The training method for the music generation model provided in the embodiments of this application includes a schematic diagram of the structure of the second generator BigVGAN in some embodiments; Figure 6 for Figure 1 A detailed flowchart of step S104; Figure 7A flowchart illustrating the steps of the training method for the music generation model provided in this application embodiment in some other embodiments; Figure 8 The training method for the music generation model provided in the embodiments of this application is illustrated in some embodiments as a generative adversarial training process. Figure 9 A flowchart illustrating the steps of the music generation method provided in some embodiments of this application; Figure 10 A schematic diagram of the vocoder structure involved in some embodiments of the music generation method provided in this application; Figure 11 A schematic diagram of the structure of a training device for a music generation model provided in an embodiment of this application; Figure 12 This is a schematic diagram of the structure of the music generation device provided in the embodiments of this application; Figure 13 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] It should be noted that although functional modules are divided in the device / system schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device / system or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0024] First, the overall concept of the training method for the music generation model provided in the embodiments of this application will be explained.

[0025] Currently, music generation technology, thanks to continuous breakthroughs in artificial intelligence algorithms, has successfully solved the core challenge of "generating complete music," achieving a leap from fragmented synthesis to complete song creation. Whether it's modeling the audio of mixed accompaniment and vocals as a whole, or modeling the accompaniment and vocals separately and then merging them to generate a complete segment, existing technologies can output high-fidelity songs. This has significantly promoted the popularization and efficiency of music creation, and is widely used in scenarios such as personal entertainment creation and commercial background music customization, becoming an important achievement of the deep integration of the music industry and artificial intelligence (AI) technology.

[0026] In existing high-fidelity song generation technologies, mainstream solutions generally use "discrete tokens" as the core modeling representation, achieving complete song generation through two typical paths: one is to directly model the mixed audio containing accompaniment and vocals using discrete tokens, outputting the fused audio segment in one go; the other is to first separate the accompaniment and vocals into independent modules, model them separately using discrete tokens, and then merge them into a complete accompaniment and vocal segment through post-processing. However, both paths have significant technical limitations—the training process of their acoustic models and vocoders heavily relies on high-sampling-rate audio data. While high-sampling-rate audio can retain richer sound details and ensure the fidelity of the final generated song, it also directly leads to a significant increase in data storage costs and computational resource consumption during the training process: on the one hand, high-sampling-rate audio data is larger, requiring more storage resources for data preprocessing and saving; on the other hand, the computational complexity of high-sampling-rate data during model training is higher, requiring higher-configuration hardware and significantly extending the training cycle.

[0027] In summary, while existing music generation technologies using discrete tokens as modeling representations can achieve high-fidelity generation of entire songs, they face the critical problem of "excessively high training costs." The strong dependence of acoustic models and vocoders on high-sampling-rate audio results in high data storage, hardware investment, and time costs during the technology's implementation. This not only increases the barrier to technological development but also limits the application and promotion of this technology by small and medium-sized teams or in resource-constrained scenarios. There is an urgent need for a music generation solution that can reduce dependence on high-sampling-rate audio and control training costs while ensuring high fidelity, thereby promoting the further popularization of the technology.

[0028] To address these shortcomings, embodiments of this application provide a training method for a music generation model, a music generation method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, aiming to overcome the deficiencies of the aforementioned related technologies. While ensuring that the music generation model outputs high-fidelity songs, it reduces the model training's dependence on high-sampling-rate audio data, thereby controlling the model training cost.

[0029] In this embodiment of the application, by obtaining the training audio encoding information output by the acoustic model in the music generation model, the first generator in the vocoder of the music generation model performs a first upsampling training on the training audio encoding information to obtain intermediate training audio data with a first sampling rate, and the second generator in the vocoder performs a second upsampling training on the intermediate training audio data to obtain training audio data with a second sampling rate, wherein the second sampling rate is greater than the first sampling rate, and the vocoder is subjected to generative adversarial training based on the training audio data to obtain the trained music generation model.

[0030] Thus, this embodiment utilizes a two-stage model structure within the vocoder. The first stage uses a first generator to perform a first upsampling training on the training audio encoding information output by the acoustic model, obtaining intermediate training audio data at a first sampling rate. This intermediate training audio data is further fed to the second stage, where a second generator performs a second upsampling training on the intermediate training audio data, resulting in training audio data at a second sampling rate, which is larger than the first sampling rate. Adversarial training is then performed based on this training audio data to obtain the trained music generation model. Compared to traditional methods where both the acoustic model and vocoder training rely on high-sampling-rate audio data, this embodiment can train the vocoder using low-sampling-rate training data output by the acoustic model at the vocoder's upstream stage. That is, it is compatible with the low-cost training results of the acoustic model using low-sampling-rate data, training the loss at different sampling rates in a multi-task manner with progressive upsampling, thereby steadily increasing the model's sampling rate step by step, ultimately training a vocoder capable of outputting high-fidelity audio. In other words, the embodiments of this application can reduce the dependence of model training on high-sampling-rate audio data while ensuring that the music generation model outputs high-fidelity songs, thereby controlling the model training cost.

[0031] Furthermore, in the embodiments of this application, when generating music, the vocoder in the trained music generation model is used to upsample the audio encoding information output by the acoustic model step by step, thereby enabling the direct output of high-fidelity audio data with a high sampling rate based on low sampling rate audio encoding information.

[0032] Next, the training method, music generation method, apparatus, electronic device, computer-readable storage medium, and computer program product of the music generation model provided in this application will be specifically described through the following embodiments, and the various detailed embodiments of the training method of the music generation model provided in this application will be described in detail first.

[0033] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0034] It should be noted that the training method for the music generation model provided in this application embodiment can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a computer device such as a smartphone, tablet, laptop, or desktop computer, or an in-vehicle terminal device (e.g., an in-vehicle computing platform) or a terminal associated with a vehicle. Associating with a vehicle means that the terminal can communicate and interact with the vehicle via a network. The server can be a backend server terminal device, configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The software can be an application implementing the training method for the music generation model, a computer program, and a storage medium carrying the computer program. It should be understood that, based on different design needs of practical applications, the terminal, server, and software that apply the training method of the music generation model provided in this application may also be other forms not listed here, and the training method of the music generation model provided in this application does not specifically limit these.

[0035] Furthermore, this application can also be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, personal computers (PCs), minicomputers, mainframe computers, vehicles, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0036] For ease of understanding and explanation, the following text will use the training method of the music generation model provided in the embodiments of this application on a terminal device as an example to describe in detail the various specific embodiments of this application. The implementation of the training method of the music generation model provided in the embodiments of this application on any other subject matter described above can refer to the process of the training method of the music generation model on a terminal device described below.

[0037] Please refer to Figure 1 , Figure 1 The flowchart illustrates the steps of the training method for the music generation model provided in this application in some embodiments. It should be understood that, although... Figure 1 The flowcharts illustrating subsequent steps show the execution order of some method steps. However, based on different design needs in practical applications, the training method for the music generation model provided in this application embodiment can, of course, employ a different execution order of method steps than shown in the figures. That is, Figure 1 The order of the method steps shown does not constitute a limitation on the execution logic order of the training method for the music generation model provided in this application embodiment. Any other order based on... Figure 1 Reasonable changes to the order of steps shown should be included within the protection scope of the training method for the music generation model provided in the embodiments of this application.

[0038] like Figure 1 As shown, in some embodiments, the terminal device can use the training method of the music generation model provided in the embodiments of this application to train the vocoder in the music generation model. The music generation model includes an acoustic model and a vocoder. The output end of the acoustic model is connected to the input end of the vocoder. The training of the vocoder may include steps S101 to S104 as shown below.

[0039] Step S101: Obtain the training audio encoding information output by the acoustic model training.

[0040] It should be noted that the training audio encoding information output by the acoustic model training can be the training audio encoding information output by the acoustic model (such as the large language model LLM or the large language model LLMA2 after pre-training and fine-tuning, etc.) for acoustic feature learning training on user input data (keywords or descriptive text and / or reference audio, etc., for requesting music type, lyrics, etc.).

[0041] During the training process of the vocoder, the terminal device first obtains the training audio encoding information output by the acoustic model of the vocoder's upper level, which learns acoustic features from user input data.

[0042] In some embodiments, the terminal device can obtain the training audio encoding information of the acoustic model's real-time training output when training the acoustic model and vocoder together.

[0043] In other embodiments, the terminal device may also collect the training audio encoding information output by the acoustic model training in advance, and then construct sample data specifically for training the vocoder using this training audio encoding information. In this way, when training the vocoder, the terminal device can extract the previously stored training audio encoding information to perform the training operation on the vocoder.

[0044] In some embodiments, the terminal device can use user input data with a low sampling rate to train the acoustic model at low cost, and then obtain the training audio encoding information of the acoustic model training output for further training of the vocoder, so that the vocoder is compatible with the output results of the low-cost training of the acoustic model, thereby better controlling the overall training cost of the music generation model.

[0045] Step S102: Based on the first generator in the vocoder, perform a first upsampling training on the training audio coding information to obtain intermediate training audio data with a first sampling rate.

[0046] It should be noted that a vocoder can be a new two-level structure that combines partial structures from two vocoder models. For example, a vocoder can be a new two-level structure that combines partial structures from the non-autoregressive vocoder model Vocos as the first generator and partial structures from the vocoder model BigVGAN, which is based on a generative adversarial network (GAN) architecture, as the second generator. The first generator, Vocos, is suitable for "fast and lightweight" real-time vocoders, facilitating edge deployment, while the second generator, BigVGAN, is a "large and powerful" general-purpose vocoder, suitable for high-fidelity, cross-scene, and zero-shot audio generation tasks.

[0047] After obtaining the training audio encoding information, the terminal device inputs the training audio encoding information into the first generator, which then performs a first upsampling training on the training audio encoding information to obtain intermediate training audio data with a first sampling rate output by the first generator.

[0048] In some embodiments, when a terminal device combines parts of the Vocos and BigVGAN vocoder models to form a new two-level structure to build a vocoder, the first generator can be adjusted using the structure of Vocos, so that the first generator Vocos differs from the original version of the Vocos model in that it reconstructs audio from the Mel spectrum, and instead performs feature learning training on the audio encoding codec information.

[0049] In some embodiments, when the terminal device performs a first upsampling training on the training audio codec information based on the first generator Vocos, the first generator Vocos can extract features from the input codec information through the backbone network structure responsible for core feature processing and transformation in the Vocos model to achieve the first upsampling training. After upsampling, the energy spectrum at a preliminary upsampling rate of 32k is predicted. Thus, the energy spectrum at the 32k sampling rate is the intermediate training audio data at the first sampling rate (the first sampling rate is 32k) output by the first generator Vocos.

[0050] Step S103: Based on the second generator in the vocoder, perform a second upsampling training on the intermediate training audio data to obtain training audio data with a second sampling rate; the second sampling rate is greater than the first sampling rate.

[0051] After obtaining intermediate training audio data at a first sampling rate by performing first upsampling training on the training audio encoding information based on the first generator, the terminal device further uses this intermediate training audio data as the training input of the second generator. Based on the second generator, it then performs second upsampling training on the intermediate training audio data to obtain training audio data at a second sampling rate as the training output of the second generator. Here, the second sampling rate is greater than the aforementioned first sampling rate.

[0052] In some embodiments, when the terminal device performs a second upsampling training on intermediate training audio data based on the second generator BigVGAN, the intermediate training audio data undergoes layer normalization processing to obtain latent variable information. This latent variable information is further input into the second generator BigVGAN for further second upsampling training, and after upsampling, 44.1k high-fidelity training audio data is predicted. Thus, this 44.1k training audio data is the training audio data with the second sampling rate output by the second generator BigVGAN (the first sampling rate is 44.1k).

[0053] In some embodiments, the terminal device uses 44.1kHz high-fidelity audio as training data. It first trains the acoustic model using this training data to output encoded codec information (training audio encoding information). Then, based on this codec information, it sequentially trains the first generator Vocos and the second generator BigVGAN. The ultimate goal is for the second generator BigVGAN to generate 44.1kHz audio. It should be understood that, depending on the design needs of different applications, the terminal device can also use 48kHz or other sampling rates of audio as training for model training.

[0054] Step S104: Perform generative adversarial training on the vocoder based on the training audio data to obtain the trained music generation model.

[0055] The terminal device can iteratively optimize the vocoder through adversarial learning. That is, after the intermediate training audio data is trained by the second upsampling based on the second generator to obtain training audio data with the second sampling rate, the terminal device further performs generative adversarial training on the vocoder based on the training audio data to obtain the trained music generation model.

[0056] In some embodiments, the output of the second generator in the vocoder can be connected to the input of the discriminator. This allows for adversarial training of the vocoder and discriminator, alternately updating them to obtain the final vocoder and thus the trained music generation model. During training, the vocoder uses the first and second generators to attempt to generate waveforms that can deceive the discriminator, which then attempts to distinguish the synthesized waveforms. The discriminator calculates feature matching loss, Mel spectrum loss, and amplitude spectrum loss under different window lengths based on the generated audio data and the real audio (also known as the target audio) to update the generator's gradient. The generator aims for a smaller loss between the generated audio data and the real audio, while the discriminator aims for a larger loss between the real audio and the generated audio. After training, the trained music generation model may only require the vocoder component.

[0057] In this embodiment, during the training of the vocoder by the terminal device, the training audio encoding information output by the acoustic model of the vocoder's parent unit, which learns acoustic features from user input data, is first obtained. Then, this training audio encoding information is input into a first generator, which performs a first upsampling training on the training audio encoding information to obtain intermediate training audio data with a first sampling rate output by the first generator. Next, this intermediate training audio data is used as the training input for a second generator, which performs a second upsampling training on the intermediate training audio data to obtain training audio data with a second sampling rate output by the second generator. Here, the second sampling rate is greater than the first sampling rate. Finally, generative adversarial training is performed on the vocoder based on this training audio data to obtain a trained music generation model.

[0058] Thus, this embodiment utilizes a two-stage model structure within the vocoder. The first stage uses a first generator to perform a first upsampling training on the training audio encoding information output by the acoustic model, obtaining intermediate training audio data at a first sampling rate. This intermediate training audio data is further fed to the second stage, where a second generator performs a second upsampling training on the intermediate training audio data, resulting in training audio data at a second sampling rate, which is larger than the first sampling rate. Adversarial training is then performed based on this training audio data to obtain the trained music generation model. Compared to traditional methods where both the acoustic model and vocoder training rely on high-sampling-rate audio data, this embodiment can train the vocoder using low-sampling-rate training data output by the acoustic model at the vocoder's upstream stage. That is, it is compatible with the low-cost training results of the acoustic model using low-sampling-rate data, training the loss at different sampling rates in a multi-task manner with progressive upsampling, thereby steadily increasing the model's sampling rate step by step, ultimately training a vocoder capable of outputting high-fidelity audio. In other words, the embodiments of this application can reduce the dependence of model training on high-sampling-rate audio data while ensuring that the music generation model outputs high-fidelity songs, thereby controlling the model training cost.

[0059] Please refer to Figure 2 , Figure 2 for Figure 1 A detailed flowchart of step S102.

[0060] like Figure 2 As shown, in some embodiments, step S102 above: performing a first upsampling training on the training audio coding information based on the first generator in the vocoder to obtain intermediate training audio data with a first sampling rate may include step S201 as shown below.

[0061] Step S201: Based on the first generator in the vocoder, the training audio encoding information is encoded into hidden layer tensor information for feature learning, and the hidden layer tensor information is subjected to first upsampling training to obtain intermediate training audio data with a first sampling rate.

[0062] When the terminal device performs first upsampling training on the training audio encoding information output by the acoustic model based on the first generator, it can first encode the training audio encoding information into hidden layer tensor information based on the first generator, then perform feature learning on the hidden layer tensor information, and then perform first upsampling training on the hidden layer tensor information to obtain intermediate training audio data with the first sampling rate.

[0063] In some embodiments, the first generator includes a first convolutional module and stacked second convolutional modules. Based on this, the step S201 above, "encoding the training audio encoding information into hidden layer tensor information for feature learning," can include the following steps: The training audio coding information is encoded into hidden layer tensor information based on the first convolution module; Feature learning processing is performed on the hidden layer tensor information based on the second convolution module, using multi-scale feature fusion.

[0064] During the first upsampling training of the training audio encoded information based on the first generator, after the training audio encoded information is input into the first generator, it first passes through the first convolutional model of the first generator. This first convolutional module can be a one-dimensional convolutional module, thereby encoding the training audio encoded information into a hidden layer tensor. Then, the hidden layer tensor further enters the second convolutional module of the first generator. This second convolutional module can be a series of stacked one-dimensional convolutional modules. The first generator then uses this second convolutional module to perform multi-scale feature fusion feature learning processing on the hidden layer tensor information.

[0065] In some embodiments, the structure of the first generator Vocos is as follows: Figure 3As shown, it includes a first convolutional module: a one-dimensional convolutional module Conv1d, and a second convolutional module: a series of stacked one-dimensional convolutional modules ConvNeXt Blocks. Based on this, the first generator first encodes the input features (training audio encoding information, i.e., the input codec information shown in the figure) into a hidden layer tensor information through the one-dimensional convolutional module Conv1d, and then uses the stacked one-dimensional convolutional modules ConvNeXtBlocks to further learn this hidden layer tensor information. At the same time, the first generator also continuously improves the modeling ability of the first generator by maintaining the local receptive field (the convolution result after the codec information is processed by Conv1d and ConvNeXt Blocks) through residual connections.

[0066] Please refer to Figure 4 , Figure 4 for Figure 1 A detailed flowchart of step S103.

[0067] like Figure 4 As shown, in some embodiments, step S103 above, which involves performing a second upsampling training on the intermediate training audio data based on the second generator in the vocoder to obtain training audio data with a second sampling rate, may include steps S401 to S403 as shown below.

[0068] Step S401: Perform layer normalization on the intermediate training audio data to obtain latent variable information.

[0069] Step S402: Input the latent variable information into the second generator in the vocoder.

[0070] Step S403: Based on the second generator, perform a second upsampling training on the latent variable information to obtain training audio data with a second sampling rate.

[0071] When the terminal device performs second upsampling training on the intermediate training audio data output from the first generator based on the second generator, it first performs layer normalization on the intermediate training audio data to obtain latent variable information corresponding to the intermediate training audio data. Then, the terminal device further inputs the latent variable information into the second generator in the vocoder. In this way, the second generator can perform second upsampling training on the latent variable information, thereby obtaining training audio data with a second sampling rate that is larger than the first sampling rate.

[0072] In some embodiments, when the terminal device performs second upsampling training on the intermediate training audio data output by the first generator based on the second generator, it can also directly input the intermediate training audio data into the second generator, so that the second generator performs layer normalization processing on the intermediate training audio data to obtain latent variable information. Then, the second generator performs second upsampling training on the latent variable information to obtain training audio data compared to the second sampling rate.

[0073] In some embodiments, the second generator includes a one-dimensional transposed convolutional model and an anti-aliasing multi-period combination module. Based on this, the training audio data includes anti-aliasing training audio data. Step S403 above: performing a second upsampling training on the latent variable information based on the second generator to obtain training audio data with a second sampling rate may include the following steps: The training audio data with a second sampling rate is obtained by performing a second upsampling training on the latent variable information based on the one-dimensional transposed convolution model, and then the training audio data is upsampled and filtered based on the anti-aliasing multi-period combination module to obtain anti-aliasing training audio data.

[0074] During the second upsampling process of intermediate training audio data by the second generator, the terminal device first performs second upsampling training on the latent variable information obtained by layer normalization of the intermediate training audio data to obtain training audio data with a second sampling rate based on a one-dimensional transposed convolution model. In addition, the second generator also performs upsampling filtering on the training audio data based on an anti-aliasing multi-period combination module to obtain anti-aliasing training audio data.

[0075] In some embodiments, the structure of the second generator BigVGAN is as follows: Figure 5As shown, the terminal device uses the first generator, Vocos, to model the latent variable information output from the low-sampling-rate training audio encoding information. Specifically, the latent variable information obtained by layer normalization (LayerNorm) of the intermediate training audio data (Vocos backbone output) is fed into BigVGAN for learning. One advantage of BigVGAN is that it can maintain high-fidelity synthesis of out-of-domain scene data without fine-tuning. BigVGAN implements the upsampling process through a one-dimensional transposed convolutional model, TransposedConvolld, which stretches the low-frequency sampling feature sequence of the latent variable information to restore it to the original sampling rate, obtaining the training audio data. Then, BigVGAN uses the anti-aliasing multi-period combination module (AMP) to perform upsampling filtering on the training audio data, achieving the desired guiding bias in the audio synthesis process. The Anti-aliasing Multi-Period Combination Module (AMP) mainly consists of the periodic activation function Snakeld, anti-aliasing low-pass filtering, and dilated convolution DilatedConvld, which enables BigVGAN to achieve artifact-free high-fidelity audio synthesis in zero-sample scenarios (such as new languages, new songs, new instruments, and noisy recordings).

[0076] In some embodiments, the terminal device can achieve the function of gradually upsampling low-sampling-rate data to generate high-fidelity audio data by fusing the first generator and the second generator. However, improving the model's capabilities still requires a training strategy to guide the model optimization direction. That is, the terminal device can optimize the first generator and the second generator in the vocoder through adversarial learning, and simultaneously optimize the discriminator used for adversarial training.

[0077] Please refer to Figure 6 , Figure 6 for Figure 1 A detailed flowchart of step S104.

[0078] like Figure 6 As shown, in some embodiments, step S104 above: performing generative adversarial training on the vocoder based on the training audio data may include steps S601 and S602 as shown below.

[0079] Step S601: Perform loss calculation on the training audio data based on a preset discriminator to obtain a first loss calculation result; the discriminator includes a multi-period discriminator, a multi-scale subband constant transformation discriminator, and a first multi-resolution spectrum discriminator, and the first loss calculation result includes the sum of the results obtained by the multi-period discriminator, the multi-scale subband constant transformation discriminator, and the first multi-resolution spectrum discriminator respectively performing loss calculation on the training audio data.

[0080] After obtaining the training audio data output by the second generator, the terminal device inputs the training audio data into the discriminator for loss calculation. The discriminator includes a multi-period discriminator, a multi-scale subband constant transformation discriminator, and a first multi-resolution spectrum discriminator. Each discriminator performs loss calculation based on the training audio data to obtain its own result. Then, the results calculated by each discriminator are added together to obtain the first loss calculation result obtained by performing loss calculation on the training audio data.

[0081] It should be noted that the terminal device uses multiple different discriminators to calculate the loss on the training audio data output by the second generator. This is based on the individual roles of these different discriminators, adversarially helping the vocoder containing the second generator to generate higher quality audio data. Among the multiple discriminators, the multi-period discriminator (MPD) can effectively capture the periodic features in the audio signal, which is very important for generating high-quality audio (especially music or speech with complex periodic structures). For example, given a period [2,3,,5,7,11], it will perform a convolution operation on the matrix [1*T]->[1,T / 2,2], [1,T / 3,3]… to capture the periodic details. Furthermore, compared to the traditional Short-Time Fourier Transform (STFT), the Multi-Scale Sub-Band Constant-Q Transformer (MS-SB-CQT) offers higher frequency resolution in the low-frequency region and higher time resolution in the high-frequency region. This dynamic resolution makes CQT more suitable for processing audio signals with complex frequency structures. The MS-SB-CQT discriminator operates the CQT spectrogram at multiple different resolutions, thereby capturing features across different frequency ranges. By setting different resolutions (e.g., by adjusting the CQT's bins per octave parameter), the discriminator can perform detailed analysis across different frequency bands. Dividing the CQT spectrogram into different sub-bands, each corresponding to a specific octave range, further refines the analysis of different frequency regions, improving the model's ability to model pitch and harmony. The first multi-resolution spectral discriminator (MRS) fundamentally solves the oversmoothing problem of vocoders in full-band audio synthesis by using multi-resolution spectral analysis and multi-scale adversarial supervision, significantly improving the realism and detail of the generated audio. The core value of MRS lies in its deep integration of the multi-scale characteristics of spectral analysis with the adversarial mechanism of GANs, providing a universal framework for high-fidelity audio synthesis. Through the alternating optimization process of the multi-period discriminator (MPD), the multi-scale sub-band constant transform discriminator (MRD), and the first multi-resolution spectral discriminator (MRS), the periodic characteristics (such as fundamental frequency and harmonic structure) in the audio modeling process can be displayed. This allows for detailed learning and optimization of mid-to-low and high frequencies during short-time Fourier transforms at different window lengths and step sizes.

[0082] Step S602: Update the parameters of the second generator based on the first loss calculation result.

[0083] After obtaining the first loss calculation result by performing loss calculation on the training audio data based on the discriminator, the terminal device further updates the parameters of the second generator based on the first loss calculation result. That is, based on the first loss calculation result, the gradient of the second generator model parameters is calculated by backpropagation (solving the partial derivative of the loss with respect to each parameter), and then the model parameters of the second generator are adjusted according to the gradient obtained by backpropagation and according to the preset update rules (such as the iterative formula of Adaptive Moment Estimation Adam and Stochastic Gradient Descent SGD).

[0084] In some embodiments, when the terminal device performs generative adversarial training on the vocoder, in addition to calculating the first loss calculation result based on the training audio data output by the second generator to update the parameters of the second generator, it can also perform loss calculation based on the intermediate training audio data output by the first generator to update the parameters of the first generator, thereby iteratively optimizing the vocoder to obtain the final vocoder, and thus obtaining the trained music generation model.

[0085] Please refer to Figure 7 , Figure 7 The training method for the music generation model provided in this application is illustrated in some other embodiments as a flowchart of the steps.

[0086] like Figure 7 As shown, in some embodiments, the training method for the music generation model provided in this application may further include steps S701 and S702 as shown below.

[0087] Step S701: Perform loss calculation on the intermediate training audio data based on a preset discriminator to obtain a second loss calculation result; the discriminator further includes a second multi-resolution spectrum discriminator, and the second loss calculation result is the result obtained by the second multi-resolution spectrum discriminator performing loss calculation on the intermediate training audio data.

[0088] It should be noted that the second multi-resolution spectrum discriminator in the discriminator is the same as the first multi-resolution spectrum discriminator mentioned above, and is a multi-resolution spectrum discriminator specifically used for intermediate training audio data.

[0089] After obtaining the intermediate training audio data output by the first generator, the terminal device can also input the intermediate training audio data into the discriminator for loss calculation. The discriminator includes a second multi-resolution spectrum discriminator, which performs loss calculation based on the intermediate training audio data to obtain a second loss calculation result.

[0090] Step S702: Update the parameters of the first generator based on the second loss calculation result.

[0091] After obtaining a second loss calculation result by performing loss calculation on the intermediate training audio data based on the discriminator, the terminal device further updates the parameters of the first generator based on this second loss calculation result. That is, it calculates the gradient of the first generator model parameters by backpropagation based on the second loss calculation result, and then adjusts the model parameters of the first generator according to the gradient obtained by backpropagation and a preset update rule.

[0092] In some embodiments, the process of the terminal device performing generative adversarial training on the vocoder based on the discriminator's loss calculation is as follows: Figure 8 As shown, when the terminal device trains the vocoder, the model input consists of audio data (audio) at the original sampling rate (e.g., 44.1k), codec information extracted from this audio data (training audio coding information output by the acoustic model training), Mel spectra of the 44.1k audio data, and power spectra of the audio data at a first sampling rate of 32k. The input codec information is trained by the first generator, vocos, through a first upsampling, outputting the predicted power spectra at the first sampling rate of 32k (intermediate training audio data). This power spectra is then normalized to obtain latent variable information, which is fed to the second generator, bigVGAN. The second generator, bigVGAN, further performs a second upsampling on this latent variable information to obtain training audio data at a second sampling rate of 44.1k, thus achieving the learning of generating high-fidelity audio. During training, an adversarial network training scheme is used, alternately optimizing the generator (including the first generator vocos and the second generator bigVGAN) and the discriminator. The discriminator comprises four types of discriminators (also known as optimizers). The multi-period discriminator (MPD), multi-scale subband constant transform discriminator (MRD), and first multi-resolution spectrum discriminator (MRS) are optimized for high-fidelity 44.1kHz audio data. The second multi-resolution spectrum discriminator (MRS_stft) optimizes the discriminator for the intermediate 32kHz audio output, simultaneously assisting the first generator (vocos) in adversarial optimization. The individual losses calculated by each discriminator are summed to form the final total loss (including the first and second loss calculations). The gradient of this total loss is then backpropagated to update the parameters of the first generator (vocos), the second generator (bigVGAN), and all discriminators.

[0093] In this embodiment, a vocoder composed of multi-level generator modules is integrated into the terminal device to learn the ability to generate upsampled high-fidelity audio. Adversarial training is used to improve the vocoder's capabilities. This not only allows the vocoder to generate high-sampling-rate audio data during training, but also enables the upper-level acoustic model of the vocoder to fully utilize low-sampling-rate data for model optimization. Thus, while ensuring the final trained music generation model outputs high-fidelity songs, the dependence of the model training process on high-sampling-rate audio data is reduced, effectively lowering the model training cost.

[0094] The following describes in detail the specific embodiments of the music generation method provided in this application.

[0095] Please refer to Figure 9 , Figure 9 The following is a flowchart illustrating the steps of the music generation method provided in some embodiments of this application.

[0096] like Figure 9 As shown, in some embodiments, the music generation method provided in this application may include steps S901 to S903 as shown below.

[0097] Step S901: Obtain the audio encoding information output by the acoustic model; the acoustic model is the acoustic model in the trained music generation model.

[0098] When a terminal device performs a music generation task using a trained music generation model, it first uses the acoustic model in the trained music generation model to learn the acoustic features of the user input data (keywords or descriptive text and / or reference audio, etc., for requesting music type, lyrics, etc.), and then outputs audio encoding information corresponding to the user input data.

[0099] Step S902: Perform a first upsampling process on the audio encoded information based on the first generator in the vocoder to obtain intermediate audio data with a first sampling rate; the vocoder is the vocoder in the trained music generation model.

[0100] During the process of the terminal device performing music generation tasks using the trained music generation model, after the acoustic model in the trained music generation model outputs audio encoding information, the audio encoding information is further input into the first generator, which performs a first upsampling process on the audio encoding information to obtain intermediate audio data with a first sampling rate output by the first generator.

[0101] Step S903: Perform a second upsampling process on the intermediate audio data based on the second generator in the vocoder to obtain audio data with a second sampling rate.

[0102] After the terminal device performs a first upsampling process on the audio encoding information based on the first generator to obtain intermediate audio data at the first sampling rate, the intermediate audio data is further input into the second generator, which then performs a second upsampling process on the intermediate audio data to obtain audio data at the second sampling rate.

[0103] In some embodiments, the structure of the vocoder in the trained music generation model is as follows: Figure 10 As shown, when the terminal device performs a music generation task using the trained music generation model, after obtaining the audio encoding information (codec) output by the acoustic model in the model, this codec information first enters the first generator, Vocos backbone, in the vocoder. The first generator, Vocos backbone, performs a first upsampling process on the audio encoding information, outputting low-sampling-rate intermediate audio data. The intermediate audio data continues to enter the second generator, bigVGAN, in the vocoder. The second generator, bigVGAN, further performs a second upsampling process on the intermediate audio data, thereby outputting high-fidelity audio data 44.1K audio.

[0104] In this embodiment, when music is generated using a trained music generation model on a terminal device, the vocoder in the trained music generation model performs progressive upsampling on the audio encoding information output by the acoustic model. This allows for the direct output of high-fidelity audio data with a high sampling rate even based on low-sampling-rate audio encoding information. Furthermore, since the acoustic model uses a large language model, it can stream-generate voice tokens. The vocoder simultaneously models and mixes the accompaniment and vocals for the input voice token, enabling the streaming output of single-track audio data (such as a song) that combines accompaniment and vocals. Compared to the traditional method of generating accompaniment and vocals separately offline and then post-processing to generate the entire song, this approach allows for better implementation of the music generation technology system. Moreover, since the vocoder can directly output songs containing both accompaniment and vocals, this embodiment can generate songs more efficiently in offline creation scenarios, improving the speed of music generation.

[0105] Please see Figure 11 This application also provides a training apparatus for a music generation model. The music generation model includes an acoustic model and a vocoder, with the output of the acoustic model connected to the input of the vocoder. The training apparatus for the music generation model provided in this application can implement the above-described training method for the music generation model.

[0106] like Figure 11As shown, the training device for the music generation model provided in this application embodiment includes a first acquisition module 1101, a first upsampling training module 1102, a second upsampling training module 1103, and an adversarial training module 1104. Among them, The first acquisition module 1101 is used to acquire the training audio encoding information output by the acoustic model training; The first upsampling training module 1102 is used to perform first upsampling training on the training audio coding information based on the first generator in the vocoder to obtain intermediate training audio data with a first sampling rate. The second upsampling training module 1103 is used to perform second upsampling training on the intermediate training audio data based on the second generator in the vocoder to obtain training audio data with a second sampling rate; the second sampling rate is greater than the first sampling rate. The adversarial training module 1104 is used to perform generative adversarial training on the vocoder based on the training audio data to obtain a trained music generation model.

[0107] In some embodiments, the first upsampling training module 1102 is further configured to encode the training audio encoding information into hidden layer tensor information for feature learning based on the first generator in the vocoder, and perform first upsampling training on the hidden layer tensor information to obtain intermediate training audio data with a first sampling rate.

[0108] In some embodiments, the first generator includes a first convolutional module and stacked second convolutional modules; the first upsampling training module 1102 is further configured to encode the training audio coding information into hidden layer tensor information based on the first convolutional module; and to perform multi-scale feature fusion feature learning processing on the hidden layer tensor information based on the second convolutional module.

[0109] In some embodiments, the second upsampling training module 1103 is further configured to perform layer normalization processing on the intermediate training audio data to obtain latent variable information; input the latent variable information into the second generator in the vocoder; and perform second upsampling training on the latent variable information based on the second generator to obtain training audio data with a second sampling rate.

[0110] In some embodiments, the second generator includes a one-dimensional transposed convolutional model and an anti-aliasing multi-period combination module; the training audio data includes anti-aliasing training audio data, and the second upsampling training module 1103 is further configured to perform second upsampling training on the latent variable information based on the one-dimensional transposed convolutional model to obtain training audio data with a second sampling rate, and perform upsampling filtering processing on the training audio data based on the anti-aliasing multi-period combination module to obtain anti-aliasing training audio data.

[0111] In some embodiments, the adversarial training module 1104 is further configured to perform loss calculation on the training audio data based on a preset discriminator to obtain a first loss calculation result; the discriminator includes a multi-period discriminator, a multi-scale subband constant transformation discriminator, and a first multi-resolution spectrum discriminator, the first loss calculation result including the sum of the results obtained by the multi-period discriminator, the multi-scale subband constant transformation discriminator, and the first multi-resolution spectrum discriminator respectively performing loss calculation on the training audio data; and to update the parameters of the second generator based on the first loss calculation result.

[0112] In some embodiments, the adversarial training module 1104 is further configured to perform loss calculation on the intermediate training audio data based on a preset discriminator to obtain a second loss calculation result; the discriminator further includes a second multi-resolution spectrum discriminator, the second loss calculation result is the result obtained by the second multi-resolution spectrum discriminator performing loss calculation on the intermediate training audio data; and update the parameters of the first generator based on the second loss calculation result.

[0113] It should be noted that the specific implementation of the training device for the music generation model provided in this application is basically the same as the specific implementation of the training method for the music generation model described above, and will not be repeated here.

[0114] Please see Figure 12 This application also provides a music generation apparatus that can implement the above-described music generation method.

[0115] like Figure 12 As shown, the music generation device provided in this embodiment includes a second acquisition module 1201, a first upsampling processing module 1202, and a second upsampling processing module 1203. Wherein, The second acquisition module 1201 is used to acquire the audio encoding information output by the acoustic model; the acoustic model is the acoustic model in the trained music generation model. The first upsampling processing module 1202 is used to perform a first upsampling process on the audio encoding information based on the first generator in the vocoder to obtain intermediate audio data with a first sampling rate; the vocoder is the vocoder in the trained music generation model. The second upsampling processing module 1203 is used to perform a second upsampling process on the intermediate audio data based on the second generator in the vocoder to obtain audio data with a second sampling rate.

[0116] It should be noted that the specific implementation of the music generation device provided in this application is basically the same as the specific implementation of the music generation method described above, and will not be repeated here.

[0117] Please see Figure 13 This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-mentioned music generation model training and / or music generation method.

[0118] In some embodiments, the electronic device can be any smart terminal such as a tablet computer, smartphone, in-vehicle hardware platform (e.g., in-vehicle computer), or wearable device.

[0119] like Figure 13 As shown, the electronic device provided in this application embodiment may include: The processor 1301 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1302 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1302 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1302, and the processor 1301 calls and executes the training method and / or music generation method of the music generation model in the embodiments of this application. The input / output interface 1303 is used to implement information input and output; The communication interface 1304 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1305 transmits information between various components of the device (e.g., processor 1301, memory 1302, input / output interface 1303, and communication interface 1304); The processor 1301, memory 1302, input / output interface 1303 and communication interface 1304 are connected to each other within the device via bus 1305.

[0120] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the training method and / or music generation method of the above-described music generation model.

[0121] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0122] This application also provides a computer program product, including a computer program. The steps implemented by the computer program when executed by a processor are basically the same as the specific embodiments of the training method and / or music generation method of the music generation model described above, and will not be repeated here.

[0123] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0124] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0125] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0127] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0128] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0129] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0130] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0132] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for training a music generation model, characterized in that, The music generation model comprises an acoustic model and a vocoder, an output end of the acoustic model being connected to an input end of the vocoder; the method comprises: obtaining training audio coding information output by the acoustic model; based on a first generator in the vocoder, first up-sampling training the training audio coding information to obtain intermediate training audio data of a first sampling rate; based on a second generator in the vocoder, second up-sampling training the intermediate training audio data to obtain training audio data of a second sampling rate; the second sampling rate is greater than the first sampling rate; based on the training audio data, generative adversarial training is performed on the vocoder to obtain a trained music generation model.

2. The method of claim 1, wherein, The method comprises: based on the first generator in the vocoder, the training audio coding information is encoded into hidden layer tensor information for feature learning, and the hidden layer tensor information is first up-sampled to obtain intermediate training audio data of a first sampling rate.

3. The method of claim 2, wherein, The first generator comprises a first convolution module and a stacked second convolution module; The method comprises: based on the first convolution module, the training audio coding information is encoded into hidden layer tensor information; based on the second convolution module, the hidden layer tensor information is subjected to multi-scale feature fusion feature learning processing.

4. The method of claim 1, wherein, The method comprises: performing layer normalization processing on the intermediate training audio data to obtain latent variable information; inputting the latent variable information into the second generator in the vocoder; based on the second generator, second up-sampling training the latent variable information to obtain training audio data of a second sampling rate.

5. The method of any one of claim 4, characterized in that, The second generator comprises a one-dimensional transposed convolution model and an anti-aliasing multi-period combination module; The training audio data comprises anti-aliasing training audio data, and the method comprises: based on the one-dimensional transposed convolution model, second up-sampling training the latent variable information to obtain training audio data of a second sampling rate, and based on the anti-aliasing multi-period combination module, up-sampling filtering processing is performed on the training audio data to obtain anti-aliasing training audio data.

6. The method of claim 1, wherein, The method comprises: perform loss calculation on the training audio data based on a preset discriminator to obtain a first loss calculation result; the discriminator includes a multi-period discriminator, a multi-scale sub-band constant transformation discriminator, and a first multi-resolution spectrum discriminator, and the first loss calculation result includes a sum of results obtained by performing loss calculation on the training audio data by the multi-period discriminator, the multi-scale sub-band constant transformation discriminator, and the first multi-resolution spectrum discriminator respectively; perform parameter updating on the second generator based on the first loss calculation result.

7. The method of claim 6, wherein, The method further includes: perform loss calculation on the intermediate training audio data based on a preset discriminator to obtain a second loss calculation result; the discriminator further includes a second multi-resolution spectrum discriminator, and the second loss calculation result is a result obtained by performing loss calculation on the intermediate training audio data by the second multi-resolution spectrum discriminator; perform parameter updating on the first generator based on the second loss calculation result.

8. A music generation method characterized by, The method includes: obtain audio encoding information output by an acoustic model; the acoustic model is an acoustic model in a trained music generation model; perform first up-sampling processing on the audio encoding information based on a first generator in a vocoder to obtain intermediate audio data at a first sampling rate; the vocoder is a vocoder in the trained music generation model; perform second up-sampling processing on the intermediate audio data based on a second generator in the vocoder to obtain audio data at a second sampling rate.

9. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the training method of the music generation model in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the training method of the music generation model in any one of claims 1 to 7.

11. A computer program product, characterised in that, The computer program product includes a computer program, and the computer program is executed by the processor to implement the training method of the music generation model in any one of claims 1 to 7.

Images