Method, device and equipment for training a generative model and storage medium

Abstract

Embodiments of the present disclosure relate to a method, device, equipment and storage medium for training a generative model. The method provided herein includes: obtaining first audio content corresponding to a first timbre and second audio content corresponding to a second timbre; processing the first audio content and the second audio content by using a first generative model to generate third audio content; providing the third audio content and a first part of the second audio content to a second generative model to generate first audio features; and training the second generative model based on the first audio features and second audio features corresponding to the second audio content. The embodiments of the present disclosure can train the second generative model based on a comparison between audio features of the reconstructed second audio content and audio features of the original second audio content, so that the trained second generative model has a better generation effect.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein relate generally to the field of computers, and more particularly to methods, apparatus, devices, and computer-readable storage media for training generative models. Background Technology

[0002] With the development of the internet and computer technology, audio feature processing has also flourished. In the field of audio feature processing, generative models have received widespread attention and use. Therefore, the generation effect of generative models has become a focal point of concern. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for training a generative model is provided. The method includes: acquiring first audio content corresponding to a first timbre and second audio content corresponding to a second timbre; processing the first and second audio content using a first generative model to generate third audio content; providing a first portion of the third and second audio content to a second generative model to generate first audio features; and training the second generative model based on the first audio features and the second audio features corresponding to the second audio content.

[0004] In a second aspect of this disclosure, an apparatus for training a generative model is provided. The apparatus includes: an acquisition module configured to acquire first audio content corresponding to a first timbre and second audio content corresponding to a second timbre; a generation module configured to process the first and second audio content using the first generative model to generate third audio content; a providing module configured to provide the third audio content and a first portion of the second audio content to the second generative model to generate first audio features; and a training module configured to train the second generative model based on the first audio features and the second audio features corresponding to the second audio content.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram is shown of an example environment in which embodiments of the present disclosure may be implemented;

[0010] Figure 2 A flowchart illustrating an example of training a generative model according to some embodiments of this disclosure is shown;

[0011] Figures 3A to 3D A flowchart illustrating an example of training a generative model according to some embodiments of this disclosure is shown;

[0012] Figure 4 A schematic structural block diagram of an example training and generative model apparatus according to some embodiments of the present disclosure is shown; and

[0013] Figure 5 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0017] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0018] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0019] The embodiments disclosed herein involve model training and inference. It is understood that the data involved in the training and inference process (including but not limited to the data itself, the acquisition or use of the data) complies with the requirements of relevant laws, regulations and related provisions.

[0020] Traditional approaches suffer from two main drawbacks. First, electronic devices cannot accurately convert the timbre of the target audio. Second, traditional models cannot convert the timbre of the target audio based on a segment of audio within a preset time period; instead, they require training with a large amount of audio to ultimately obtain the target audio for timbre conversion.

[0021] Embodiments of this disclosure propose a scheme for training a generative model. According to this scheme, a first audio content corresponding to a first timbre and a second audio content corresponding to a second timbre can be obtained; the first audio content and the second audio content can be processed using a first generative model to generate third audio content; the third audio content and a first portion of the second audio content can be provided to a second generative model to generate first audio features; and the second generative model can be trained based on the first audio features and the second audio features corresponding to the second audio content.

[0022] The embodiments of this disclosure can train a second generative model by comparing the audio features of the reconstructed second audio content with the audio features of the original second audio content, thereby enabling the trained second generative model to have better generation performance.

[0023] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0024] Example Environment

[0025] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, the example environment 100 may include an electronic device 110 and a target model 120.

[0026] In this example environment 100, electronic device 110 can use first audio content and second audio content as input to train target model 120 (e.g., a second generative model). In some embodiments, electronic device 110 is at least configured to process the received input content based on the first generative model to generate third audio content. Further, electronic device 110 can construct training data based on the third audio content and a first portion of the second audio content, and electronic device 110 trains the target model 120 based on the obtained training data.

[0027] In some embodiments, the electronic device 110 may establish a communication connection with the target model 120. That is, the electronic device 110 may invoke the local or remote target model 120.

[0028] In some embodiments, electronic device 110 may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, electronic device 110 may also support any type of user-facing interface (such as "wearable" circuitry).

[0029] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0030] Example process

[0031] Figure 2 A flowchart of an example process 200 for training a generative model according to some embodiments of the present disclosure is shown. Process 200 can be implemented at electronic device 110. Reference is made below. Figure 1 To describe process 200.

[0032] like Figure 2 As shown in box 210, electronic device 110 acquires the first audio content corresponding to the first timbre and the second audio content corresponding to the second timbre.

[0033] Figure 3A A schematic diagram 300A illustrates an example of training a generative model according to some embodiments of the present disclosure. For example... Figure 3A As shown, the electronic device 110 can acquire first audio content 310 and second audio content 311. As an example, such first audio content 310 may be song A, and such second audio content 311 may be song B with a different timbre than song A.

[0034] Reference Figure 2 In box 220, electronic device 110 uses a first generative model to process the first audio content and the second audio content to generate the third audio content.

[0035] In some embodiments, such a first generative model may be, for example, a diffusion model.

[0036] To facilitate understanding, the training process of the first generative model will be described below.

[0037] refer to Figure 3BThe electronic device 110 can acquire a fourth audio content 321, such as audio content to be converted (e.g., song C). Furthermore, the electronic device 110 can obtain a first set of audio tokens 322 corresponding to the fourth audio content 321 based on a word segmentation encoder (e.g., a token generator).

[0038] In some embodiments, the electronic device 110 may also acquire a second portion 323 of the fourth audio content 321, such second portion being, for example, the audio content of the fourth audio content within a preset time period (e.g., 10 to 15 seconds). Further, the electronic device 110 may obtain a first encoded representation 324 corresponding to the second portion 323 based on a vocoder encoder (e.g., an audio encoder).

[0039] In some embodiments, the electronic device 110 may provide the first set of audio tokens 322 and the first encoded representation 324 obtained above to the first generative model 325 to be trained for processing in order to generate a third audio feature 326.

[0040] In some embodiments, the electronic device 110 may also encode the fourth audio content 321 based on the vocoder encoder 327 to obtain the fourth audio feature 328.

[0041] Furthermore, the electronic device 110 can compare the obtained third audio feature 326 with the fourth audio feature 328 to determine the training loss, thereby training the first generative model 325 to be trained.

[0042] In some embodiments, such training loss may be determined, for example, based on a loss function (e.g., cross-entropy loss function) or other means.

[0043] In this way, the electronic device 110 can reconstruct the audio features of the fourth audio content 321 based on the fourth audio content 321 and the fourth audio content 323 within a preset time period. Furthermore, the electronic device 110 can compare the reconstructed audio features of the fourth audio content with the original audio features of the fourth audio content 321, thereby more effectively training the first generative model and making the generated effect of the trained first generative model closer to the real effect.

[0044] For ease of understanding, the first generative model trained above will be used to process the acquired first audio content 310 and second audio content 311.

[0045] Continue to refer to Figure 3A and Figure 3C Understandable Figure 3C Yes Figure 3AA detailed description of boxes 310 to 313 is provided below. Please refer to the following text for further details. Figure 3C right Figure 3A The above content will be described.

[0046] refer to Figure 3C The device 110 can use the pre-trained first generative model 312 to process the first audio content 310 and the second audio content 311 to obtain a fifth audio feature 330. Further, the device 110 can decode the fifth audio feature 330 based on the vocoder decoder 331 to obtain a third audio content 313. In some embodiments, the text similarity between the third audio content 313 and the second audio content 311 is greater than a first threshold. The timbre similarity between the third audio content 313 and the first audio content 310 is greater than a second threshold.

[0047] As an example, electronic device 110 can process two songs A (e.g., first audio content) and B (e.g., second audio content) with different timbres based on a pre-trained first generative model 312 to obtain a reconstructed song B' (e.g., third audio content). The timbre of such reconstructed song B' is close to that of song A, while the melody and lyrics are close to those of song B.

[0048] In some embodiments, the electronic device 110 may assess the text similarity between the second audio content 311 and the third audio content 313 based on word error rate. The electronic device 110 may also assess the timbre similarity between the first audio content 310 and the third audio content 313 based on automatic speaker verification.

[0049] In some embodiments, the electronic device 110 may use the generation result of the pre-trained first generative model 312 as the input for the next iteration, so that the electronic device 110 can generate a large number of paired datasets (e.g., song B and song B', song B' and song B'', etc.) based on the pre-trained first generative model 312.

[0050] In this way, the electronic device 110 can train the second generative model multiple times based on the large amount of paired datasets obtained above. Furthermore, the timbre of the paired data generated by the electronic device 110 in the above manner will become increasingly similar, thereby improving the quality of the training data provided by the electronic device 110 to the second generative model, and thus the trained second generative model has better generation performance.

[0051] For ease of description, the following explanation will use the first round of training of the second generative model as an example.

[0052] Reference Figure 2In box 230, electronic device 110 provides a third audio content and a first portion of the second audio content to a second generation model to generate a first audio feature.

[0053] In some embodiments, such a second generative model may be, for example, a diffusion model.

[0054] Continue to refer to Figure 3A The electronic device 110 can generate a second set of audio tokens 314 corresponding to the third audio content 313 (e.g., the reconstructed song B') based on a word segmentation encoder. In some embodiments, the electronic device 110 can also obtain a first portion 315 of the second audio content 311, such that the first portion 315 may be, for example, song B within a preset time period (e.g., 10 to 15 seconds). Further, the electronic device 110 can utilize a vocoder encoder to generate a second encoded representation 316 of the first portion 315.

[0055] Furthermore, the electronic device 110 can provide the second set of audio tokens 314 and the second encoded representation 316 to the second generative model 317 to be trained for processing to obtain the first audio feature 318. As an example, such the first audio feature 318 may be an audio feature obtained by restoring the timbre features of the reconstructed song B' based on the timbre features of an original song B.

[0056] In some embodiments, the electronic device 110 may provide the second audio content 311 to the vocoder encoder 319 for encoding the second audio content 311 to obtain a second audio feature 320. As an example, such a second audio feature 320 may be, for example, the audio features of the original song B.

[0057] Reference Figure 2 In box 240, electronic device 110 trains a second generative model based on the first audio features and the second audio features corresponding to the second audio content.

[0058] Continue to refer to Figure 3A The electronic device 110 can compare the first audio feature 318 obtained above with the second audio feature 320 to determine the training loss, thereby training the second generative model.

[0059] In this way, the electronic device 110 can compare the reconstructed audio features of song B with the original audio features of song B, thereby enabling the trained second diffusion model to have a better generation effect and improving the accuracy of the generation results.

[0060] For ease of understanding, the timbre reconstruction process of the pre-trained second generative model will be described below.

[0061] refer to Figure 3DThe electronic device 110 can acquire reference audio content 341 and cue audio content 342. As an example, such reference audio content 341 may include complete reference music content, such as a complete song D. Such cue audio content 342 may, for example, be audio content with a specific timbre, such as humming.

[0062] In some embodiments, the electronic device 110 can separate the background music portion 343 and the vocal portion 344 from the reference music content 341 based on a music source separation model. In this way, the electronic device 110 can ensure that the integrity of the background music portion 343 is not affected during model processing.

[0063] In some embodiments, the electronic device 110 can generate a set of corresponding audio tokens 345 from the human voice portion 344 based on a word segmentation encoder. Further, the electronic device 110 can also generate a corresponding encoded representation 346 from the prompt audio content 342 based on a vocoder encoder.

[0064] In some embodiments, the electronic device 110 may provide the acquired audio token 345 and the encoded representation 346 to a pre-trained second generative model 347 for processing to obtain corresponding audio features 348. As an example, such audio features 348 are reconstructed song D', meaning that the reconstructed song D' has a timbre close to the cue audio content and a melody and lyrics close to the reference audio content (e.g., song D). Further, the electronic device 110 may process the audio features 348 based on the vocoder decoder 349 to generate target audio content.

[0065] Furthermore, the electronic device 110 can combine the target audio content and the background portion 343 in the reference music content to generate the target music content 350.

[0066] In this way, the embodiments of this disclosure can train a second generation model by comparing the audio features of the reconstructed second audio content with the audio features of the original second audio content, thereby enabling the trained second generation model to have better generation performance and improve the accuracy of the generation results.

[0067] Example devices and equipment

[0068] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4 A schematic structural block diagram of an apparatus 400 for example training a generative model according to certain embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or included in an electronic device 110. Various modules / components in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0069] like Figure 4 As shown, the device 400 includes an acquisition module 410 configured to acquire a first audio content corresponding to a first timbre and a second audio content corresponding to a second timbre; a generation module 420 configured to process the first audio content and the second audio content using a first generation model to generate a third audio content; a providing module 430 configured to provide the third audio content and a first portion of the second audio content to a second generation model to generate a first audio feature; and a training module 440 configured to train a second generation model based on the first audio feature and the second audio feature corresponding to the second audio content.

[0070] In some embodiments, the first generative model is trained based on the following process: providing the first generative model with fourth audio content and a second part of the fourth audio content to generate a third audio feature; and training the first generative model based on the third audio feature and the fourth audio feature corresponding to the fourth audio content.

[0071] In some embodiments, the first generative model is further trained based on the following process: using an audio encoder to generate a first encoded representation of the second part; using a token generator to generate a first set of audio tokens corresponding to the fourth audio content; and providing the first encoded representation and the first set of audio tokens to the first generative model.

[0072] In some embodiments, the generation module 420 is further configured to process the first audio content and the second audio content using a first generation model to generate a fifth audio feature; and to process the fifth audio feature using an audio decoder to generate a third audio content.

[0073] In some embodiments, the providing module 430 is further configured to use an audio encoder to generate a second encoded representation of the first part; use a token generator to generate a second set of audio tokens corresponding to the third audio content; and provide the second encoded representation and the second set of audio tokens to the second generation model.

[0074] In some embodiments, the text similarity between the third audio content and the second audio content is greater than a first threshold; and / or the timbre similarity between the third audio content and the first audio content is greater than a second threshold.

[0075] In some embodiments, the apparatus 400 further includes a content generation module configured to acquire cue audio content and reference audio content; and to provide the cue audio content and reference audio content to a second generation model to generate target audio content.

[0076] In some embodiments, the reference audio content includes a vocal portion separated from the reference music content, and the apparatus 400 further includes a content combining module configured to combine the target audio content and the background portion of the reference music content to generate the target music content.

[0077] In some embodiments, the reference audio content includes complete reference music content.

[0078] In some embodiments, the first generative model and / or the second generative model is a diffusion model.

[0079] In some embodiments, the apparatus 400 further includes a feature generation module configured to generate second audio features of the second audio content using an audio encoder.

[0080] The modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the modules in device 400 can be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0081] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to achieve Figure 1 Electronic devices 110.

[0082] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0083] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0084] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0085] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0086] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0087] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0088] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0089] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0090] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0092] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for training a generative model, comprising: Get the first audio content corresponding to the first timbre and the second audio content corresponding to the second timbre; The first audio content and the second audio content are processed using a first generative model to generate the third audio content; The third audio content and a first portion of the second audio content are provided to the second generation model to generate a first audio feature; as well as The second generation model is trained based on the first audio feature and the second audio feature corresponding to the second audio content. The provision of the third audio content and a first portion of the second audio content to the second generation model to generate the first audio feature includes: using an audio encoder to generate a second encoded representation of the first portion; using a token generator to generate a second set of audio tokens corresponding to the third audio content; and providing the second encoded representation and the second set of audio tokens to the second generation model. Wherein, the text similarity between the third audio content and the second audio content is greater than a first threshold; and / or the timbre similarity between the third audio content and the first audio content is greater than a second threshold.

2. The method of claim 1, wherein the first generative model is trained based on the following process: Provide the first generative model with fourth audio content and a second portion of the fourth audio content to generate a third audio feature; and The first generative model is trained based on the third audio feature and the fourth audio feature corresponding to the fourth audio content.

3. The method of claim 2, wherein providing the first generation model with the fourth audio content and the second portion of the fourth audio content comprises: Using an audio encoder, a first coded representation of the second part is generated; Using a token generator, a first set of audio tokens corresponding to the fourth audio content is generated; as well as The first encoded representation and the first set of audio tokens are provided to the first generative model.

4. The method of claim 1, wherein processing the first audio content and the second audio content using a first generation model to generate the third audio content comprises: The first audio content and the second audio content are processed using a first generative model to generate a fifth audio feature; as well as The fifth audio feature is processed using an audio decoder to generate the third audio content.

5. The method according to claim 1, further comprising: Obtain the prompt audio content and reference audio content; The prompt audio content and the reference audio content are provided to the second generation model to generate the target audio content.

6. The method of claim 5, wherein the reference audio content includes a human voice portion separated from the reference music content, and the method further comprises: The target audio content and the background portion of the reference music content are combined to generate the target music content.

7. The method of claim 5, wherein the reference audio content includes complete reference music content.

8. The method according to claim 1, wherein the first generation model and / or the second generation model is a diffusion model.

9. The method according to claim 1, further comprising: The second audio feature of the second audio content is generated using an audio encoder.

10. An apparatus for training a generative model, comprising: The acquisition module is configured to acquire the first audio content corresponding to the first timbre and the second audio content corresponding to the second timbre; The generation module is configured to process the first audio content and the second audio content using a first generation model to generate a third audio content; A providing module is configured to provide the third audio content and a first portion of the second audio content to a second generation model to generate a first audio feature; as well as The training module is configured to train the second generative model based on the first audio feature and the second audio feature corresponding to the second audio content; The providing module is further configured to: generate a second encoded representation of the first part using an audio encoder; and generate a second set of audio tokens corresponding to the third audio content using a token generator. And provide the second encoded representation and the second set of audio tokens to the second generative model; Wherein, the text similarity between the third audio content and the second audio content is greater than a first threshold; and / or the timbre similarity between the third audio content and the first audio content is greater than a second threshold.

11. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 9.

Images