Method and device for training music compression system, equipment and storage medium

Abstract

The embodiment of the invention relates to a method and device for training a music compression system, equipment and a storage medium. The method comprises the following steps: acquiring a first coded representation associated with training music content; processing the first encoded representation with a discrete encoder to generate a first set of discrete features corresponding to the first music data and a second set of discrete features corresponding to the second music data; using a first discrete decoder to decode the first set of discrete features to obtain a first audio feature corresponding to the first music data, and using a second discrete decoder to decode the second set of discrete features to obtain a second audio feature corresponding to the second music data; and determining a training loss based on the first audio feature, the second audio feature, and the training music content to adjust parameters of the discrete encoder and the discrete decoder based on the training loss.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and more particularly to methods, apparatus, devices, and computer-readable storage media for training music compression systems. Background Technology

[0002] With the continuous advancement of deep learning technology, the automatic composition and generation of music has gradually moved from theoretical research to practical applications. Encoding music content into discrete features is a crucial step in various music processing tasks. The quality of these discrete features directly impacts the results of the music processing task. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for training a music compression system is provided. The method includes: acquiring a first coded representation associated with training music content; processing the first coded representation using a discrete encoder to generate a first set of discrete features corresponding to first music data and a second set of discrete features corresponding to second music data; decoding the first set of discrete features using a first discrete decoder to obtain first audio features corresponding to the first music data, and decoding the second set of discrete features using a second discrete decoder to obtain second audio features corresponding to the second music data; and determining a training loss based on the first audio features, the second audio features, and the training music content, to adjust the parameters of the discrete encoder and the discrete decoder based on the training loss.

[0004] In a second aspect of this disclosure, an apparatus for training a music compression system is provided. The apparatus includes: an acquisition module configured to acquire a first coded representation associated with training music content; an encoding module configured to process the first coded representation using a discrete encoder to generate a first set of discrete features corresponding to first music data and a second set of discrete features corresponding to second music data; a decoding module configured to decode the first set of discrete features using a first discrete decoder to obtain a first audio feature corresponding to the first music data, and to decode the second set of discrete features using a second discrete decoder to obtain a second audio feature corresponding to the second music data; and a determination module configured to determine a training loss based on the first audio feature, the second audio feature, and the training music content, to adjust the parameters of the discrete encoder and the discrete decoder based on the training loss.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram of an example music compression system according to some embodiments of the present disclosure is shown;

[0010] Figure 2 A flowchart illustrating an example process for training a music compression system according to some embodiments of this disclosure is shown;

[0011] Figures 3A to 3B A schematic diagram of an example training process according to some embodiments of the present disclosure is shown;

[0012] Figure 4 A schematic structural block diagram of an apparatus for training a music compression system according to some embodiments of the present disclosure is shown; and

[0013] Figure 5 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0017] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. In this paper, "model" may also be referred to as a "machine learning model," a "machine learning network," or simply a "network," and these terms are used interchangeably. A model can also include different types of processing units or networks.

[0018] As used herein, a “unit,” “operation unit,” or “subunit” can consist of any suitable machine learning model or network. As used herein, a set of elements or similar expressions can include one or more such elements. For example, “a set of convolutional units” can include one or more convolutional units.

[0019] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0020] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0021] In this specification and the embodiments, if the training and inference of the model are involved, the data involved (including but not limited to the data itself, the acquisition and / or use of the data) comply with the requirements of relevant laws and regulations.

[0022] Encoding music content into discrete features is a crucial step in various music processing tasks. The quality of these discrete features directly impacts the results of various music processing tasks (such as music generation tasks).

[0023] In view of this, embodiments of the present disclosure propose a scheme for training a music compression system. The scheme includes: acquiring a first coded representation associated with training music content; processing the first coded representation using a discrete encoder to generate a first set of discrete features corresponding to first music data and a second set of discrete features corresponding to second music data; decoding the first set of discrete features using a first discrete decoder to obtain a first audio feature corresponding to the first music data, and decoding the second set of discrete features using a second discrete decoder to obtain a second audio feature corresponding to the second music data; and determining a training loss based on the first audio feature, the second audio feature, and the training music content, so as to adjust the parameters of the discrete encoder and the discrete decoder based on the training loss.

[0024] Therefore, by decoupling the features corresponding to different types of music data, the embodiments of this disclosure can improve the quality of audio compression and ensure the high fidelity of music signals and rich musical expressiveness.

[0025] Example System

[0026] Figure 1 A schematic diagram of a music compression system 100 according to some embodiments of the present disclosure is shown. The system 100 may be deployed in or implemented using suitable electronic equipment.

[0027] In some embodiments, the electronic device may include various types of computing systems / servers capable of providing computing power, and the electronic device may include terminal devices. Such terminal devices may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. The electronic device may, for example, include various types of computing systems / servers capable of providing computing power, such as mainframes, edge computing nodes, computing devices in cloud environments, virtual machines, etc. Although shown as a single device, the electronic device may include multiple physical devices.

[0028] In some embodiments, system 100 may also be referred to as a tokenizer, which can be used to process received music content and generate a set of audio codes for the music content.

[0029] like Figure 1 As shown, system 100 may include an audio encoder 120, a discrete encoder 130, and a discrete decoder 150. During the training of system 100, audio encoder 120 may process input audio 110 to generate an input coded representation.

[0030] Furthermore, the discrete encoder 130 can process the encoded representation to generate the hidden state 140, and can perform a vector quantization process on the hidden state 140.

[0031] The discrete decoder 150 can decode the quantization result of the hidden state 140 to generate an output coded representation. Furthermore, the system 100 can determine various types of losses for training the system 100 based on the output coded representation to adjust the parameters of the discrete encoder 130 and the discrete decoder 150 in the system 100.

[0032] The above illustrates the overall framework of system 100. As will be detailed below, system 100 may also include multiple discrete decoders to achieve encoding decoupling for different types of music data (e.g., vocal data and accompaniment data).

[0033] Example process

[0034] Figure 2A flowchart illustrating an example process 200 for training a music compression system 100 according to some embodiments of the present disclosure is shown. Process 200 can be implemented at system 100. Reference is made below. Figure 1 To describe process 200.

[0035] As shown in the figure, in box 210, system 100 acquires a first encoded representation associated with the training music content.

[0036] The following will combine Figure 3A and Figure 3B To describe the specific process of training system 100. Figure 3A An example training process according to some embodiments of this disclosure is shown.

[0037] by Figure 3A As an example, system 100 can use audio encoder 304 to encode training music content 302 to generate a first coded representation.

[0038] In some embodiments, to improve the locality of audio coding, the audio encoder 304 may be implemented based on a convolutional model, for example. By utilizing a convolutional model to implement audio coding, embodiments of this disclosure can provide coding stability.

[0039] In other embodiments, such as Figure 3B As shown, system 100 can also first perform track-splitting processing on the training music content 322. Specifically, system 100 can decompose the training music content 322 into first audio content corresponding to the first music data (e.g., vocal data), such as vocal track 324. System 100 can also decompose the training music content 322 into second audio content corresponding to the second music data (e.g., accompaniment data), such as accompaniment track 326. It should be understood that, as mentioned above, the training music content 322 (including but not limited to itself, its acquisition, and / or use) complies with the requirements of applicable laws and regulations.

[0040] Furthermore, system 100 can use audio encoder 328 to encode the first audio content (i.e., vocal track 324) to generate a first intermediate encoded representation. System 100 can also use audio encoder 330 to encode the second audio content (i.e., accompaniment track 326) to generate a second intermediate encoded representation.

[0041] Furthermore, system 100 can generate a first encoded representation to be processed by discrete encoder 332 based on a first intermediate encoded representation and a second intermediate encoded representation. For example, system 100 can provide the first intermediate encoded representation and the second intermediate encoded representation to discrete encoder 332 in sequence. Alternatively, system 100 can also construct the first encoded representation to be provided to discrete encoder 332 by combining the first intermediate encoded representation and the second intermediate encoded representation.

[0042] Continue to refer to Figure 2 In block 220, system 100 uses the discrete encoder to process the first encoded representation to generate a first set of discrete features corresponding to the first music data and a second set of discrete features corresponding to the second music data.

[0043] Continue to refer to Figure 3A For example, system 100 can use discrete encoder 306 to convert the received first encoded representation into a second encoded representation, such as hidden state 308.

[0044] Furthermore, system 100 can quantize the second encoded representation (e.g., hidden state 308) into the first set of discrete features based on a first portion of the target codebook associated with the discrete encoder. As an example, such as Figure 3A As shown, system 100 can perform a vector quantization process related to human voice features based on the first part of the target codebook.

[0045] like Figure 3A As shown, system 100 can also quantize the second encoded representation (e.g., hidden state 308) into the second set of discrete features based on a second portion of the target codebook associated with the discrete encoder. As an example, such as Figure 3A As shown, system 100 can perform a vector quantization process related to accompaniment features based on the second part of the target codebook.

[0046] In some embodiments, the two parts of the target codebook can be used to maintain vector representations related to vocal features and feature representations related to accompaniment features, respectively. Such a codebook structure can also be called a conjoined dual-codebook.

[0047] for Figure 3B Examples, similar to Figure 3A In the described process, system 100 can use discrete encoder 332 to convert the received first encoded representation into a second encoded representation, such as hidden state 334.

[0048] Furthermore, system 100 can quantize the second encoded representation (e.g., hidden state 334) into the first set of discrete features based on a first portion of the target codebook associated with the discrete encoder. As an example, such as Figure 3B As shown, system 100 can perform a vector quantization process related to human voice features based on the first part of the target codebook.

[0049] like Figure 3B As shown, system 100 can also quantize the second encoded representation (e.g., hidden state 334) into the second set of discrete features based on a second portion of the target codebook associated with the discrete encoder. As an example, such as Figure 3B As shown, system 100 can perform a vector quantization process related to accompaniment features based on the second part of the target codebook.

[0050] In some embodiments, for Figure 3B For example, in generating the first set of discrete features (i.e., discrete features corresponding to the voice data), system 100 may consider only the encoded representation related to the voice track 324. In generating the second set of discrete features (i.e., discrete features corresponding to the accompaniment data), system 100 may consider only the encoded representation related to the accompaniment track 326.

[0051] Continue to refer to Figure 2 In box 230, system 100 uses the first discrete decoder to decode the first set of discrete features to obtain the first audio feature corresponding to the first music data, and uses the second discrete decoder to decode the second set of discrete features to obtain the second audio feature corresponding to the second music data.

[0052] by Figure 3A As an example, system 100 may include a voice discrete decoder 310 (also referred to as a first discrete decoder) and an accompaniment discrete decoder 312 (also referred to as a second discrete decoder).

[0053] Furthermore, the human voice discrete decoder 310 can decode the first set of discrete features generated based on the human voice vector quantization process to generate the corresponding human voice audio features (also known as the first audio features).

[0054] The accompaniment discrete decoder 312 can decode the second set of discrete features generated based on the accompaniment vector quantization process to generate the corresponding accompaniment audio features (also known as the second audio features).

[0055] for Figure 3B For example, system 100 may include a voice discrete decoder 336 (also known as a first discrete decoder) and an accompaniment discrete decoder 338 (also known as a second discrete decoder).

[0056] Similarly, the human voice discrete decoder 336 can decode the first set of discrete features generated based on the human voice vector quantization process to generate the corresponding human voice audio features (also known as the first audio features).

[0057] The accompaniment discrete decoder 338 can decode the second set of discrete features generated based on the accompaniment vector quantization process to generate the corresponding accompaniment audio features (also known as the second audio features).

[0058] Additionally, such as Figure 3B As shown, system 100 may also include a hybrid audio discrete decoder 340 (also known as a third discrete decoder).

[0059] System 100 can construct a third set of discrete features based on the first set of discrete features and the second set of discrete features. For example, system 100 can mix the first set of discrete features and the second set of discrete features so that the constructed third set of discrete features can simultaneously represent vocal data and accompaniment data.

[0060] Furthermore, the hybrid audio discrete decoder 340 can decode the third set of discrete features to generate a third audio feature. The third audio feature can correspond to the content of the hybrid audio, that is, it includes both vocal content and accompaniment content.

[0061] Continue to refer to Figure 2 In box 240, system 100 determines a training loss based on the first audio feature, the second audio feature, and the training music content, in order to adjust the parameters of the discrete encoder and the discrete decoder based on the training loss.

[0062] like Figure 3A As shown, system 100 can determine a first set of losses 314 related to the voice data based on the first audio features output by voice discrete decoder 310. System 100 can determine a second set of losses 316 related to the accompaniment data based on the second audio features output by accompaniment discrete decoder 312.

[0063] In some embodiments, the first set of losses 314 and / or the second set of losses 316 may include audio reconstruction losses, which can be used to characterize the feature differences between the reconstructed audio signal based on the first audio feature or the second audio feature and the original audio signal.

[0064] In some embodiments, the first set of losses 314 and / or the second set of losses 316 may include timbre loss, which can be used to characterize the timbre difference between the audio content reconstructed based on the first audio feature or the second audio feature and the reference music content.

[0065] In some embodiments, the first set of losses 314 and / or the second set of losses 316 may include speech-related losses, which can be used to characterize the difference between the text and / or phonemes identified based on the first audio feature voice content and the text and / or phonemes corresponding to the reference music content.

[0066] In some embodiments, the first set of losses 314 and / or the second set of losses 316 may include pitch reconstruction losses, which can be used to characterize the pitch difference between the audio content reconstructed based on the first audio feature or the second audio feature and the reference music content.

[0067] In some embodiments, the first set of losses 314 and / or the second set of losses 316 may include perceptual reconstruction losses, which can be used to characterize the difference between the audio content reconstructed based on the first audio feature or the second audio feature and the reference music content at the perceptual level (the naturalness of the music content).

[0068] In some embodiments, the first set of losses 314 and / or the second set of losses 316 may include adversarial reconstruction losses, which may be used to characterize the loss determined by the discriminator in processing the audio content reconstructed based on the first audio feature or the second audio feature and the reference music content.

[0069] In some embodiments, the first set of losses 314 and / or the second set of losses 316 may include spectral reconstruction losses, which can be used to characterize the spectral differences between the audio content reconstructed based on the first audio feature or the second audio feature and the reference music content.

[0070] Accordingly, the system 100 can determine the final training loss based on the first set of losses 314 and the second set of losses 316, thereby adjusting the parameters of the discrete encoder 306, the vocal discrete decoder 310 and the accompaniment discrete decoder 312 in the system 100.

[0071] In other embodiments, for Figure 3B For example, system 100 can determine a first set of losses 342 related to the voice data based on a first audio feature output by voice discrete decoder 336. System 100 can determine a second set of losses 344 related to the accompaniment data based on a second audio feature output by accompaniment discrete decoder 338. Additionally, system 100 can also determine a third set of losses 346 related to the mixed audio data based on a third audio feature output by mixed audio discrete encoder 340.

[0072] In some embodiments, the loss types of the first group of losses 342, the second group of losses 344 and / or the third group of losses 346 may be the same as those of the first group of losses 314 and / or the second group of losses 316 discussed above, and will not be repeated here.

[0073] Accordingly, the system 100 can determine the final training loss based on the first set of losses 342, the second set of losses 344, and the third set of losses 346, thereby adjusting the parameters of the discrete encoder 332, the voice discrete decoder 336, the accompaniment discrete decoder 338, and the mixed audio discrete decoder 340 in the system 100.

[0074] In some embodiments, although the above description of the process of decoupling music data based on different types during compression uses vocal data and accompaniment data as examples, the embodiments of this disclosure can also be applied to other types of music data, such as drum data, data of different instruments, etc.

[0075] In some embodiments, the system 100 mentioned above can complete training based on a single training phase, without performing multiple training phases such as self-supervised learning, supervised fine-tuning, and vector quantization-based supervised fine-tuning.

[0076] In some embodiments, after the audio compression system has been trained, the audio compression system 100 can use the audio encoder and the corresponding discrete encoder to process the target music content to generate a set of audio coded representations.

[0077] Based on the process described above, by decoupling the features corresponding to different types of music data, the embodiments of this disclosure can improve the quality of audio compression and ensure the high fidelity of music signals and rich musical expressiveness.

[0078] Example devices and equipment

[0079] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4 A schematic structural block diagram of an apparatus 400 for training a music compression system according to certain embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or included in system 100. The various modules / components in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0080] like Figure 4As shown, the apparatus 400 includes: an acquisition module 410 configured to acquire a first encoded representation associated with training music content; an encoding module 420 configured to process the first encoded representation using a discrete encoder to generate a first set of discrete features corresponding to first music data and a second set of discrete features corresponding to second music data; a decoding module 430 configured to decode the first set of discrete features using a first discrete decoder to obtain a first audio feature corresponding to the first music data, and to decode the second set of discrete features using a second discrete decoder to obtain a second audio feature corresponding to the second music data; and a determination module 440 configured to determine a training loss based on the first audio feature, the second audio feature, and the training music content, so as to adjust the parameters of the discrete encoder and the discrete decoder based on the training loss.

[0081] In some embodiments, the encoding module 420 is further configured to: convert a first encoded representation into a second encoded representation using a discrete encoder; quantize the second encoded representation into a first set of discrete features based on a first portion of a target codebook associated with the discrete encoder; and quantize the second encoded representation into a second set of discrete features based on a second portion of a target codebook associated with the discrete encoder.

[0082] In some embodiments, the acquisition module 410 is further configured to: decompose the training music content into a first audio content corresponding to the first music data and a second audio content corresponding to the second music data; encode the first audio content using a first audio encoder to generate a first intermediate encoded representation; encode the second audio content using a second audio encoder to generate a second intermediate encoded representation; and determine the first encoded representation based on the first intermediate encoded representation and the second intermediate encoded representation.

[0083] In some embodiments, the music compression system further includes a third discrete decoder, and the decoding module 430 is further configured to: construct a third set of discrete features based on a first set of discrete features and a second set of discrete features; and decode the third set of discrete features using the third set of discrete decoders to generate a third audio feature.

[0084] In some embodiments, the training loss is also determined based on a third audio feature.

[0085] In some embodiments, the training loss includes at least one of the following: pitch reconstruction loss, perceptual reconstruction loss, and adversarial reconstruction loss.

[0086] In some embodiments, the first encoding representation is generated by an audio encoder, which is a convolutional model.

[0087] In some embodiments, the first music data is human voice data, and the second music data is accompaniment data.

[0088] In some embodiments, the apparatus 400 further includes a processing module configured to process the target music content using a trained audio compression system to generate a set of audio tokens.

[0089] The modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the modules in device 400 can be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0090] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to achieve Figure 1 System 100.

[0091] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0092] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0093] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0094] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0095] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0096] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0097] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0098] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0099] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0100] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0101] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method of training a music compression system, the music compression system comprising a discrete encoder, a first discrete decoder, and a second discrete decoder, the method comprising: obtaining a first encoded representation associated with training music content; processing the first encoded representation with the discrete encoder to generate a first set of discrete features corresponding to first music data and a second set of discrete features corresponding to second music data; decoding the first set of discrete features with the first discrete decoder to obtain first audio features corresponding to the first music data and decoding the second set of discrete features with the second discrete decoder to obtain second audio features corresponding to the second music data; and determining a training loss based on the first audio features, the second audio features, and the training music content to adjust parameters of the discrete encoder and the discrete decoder based on the training loss.

2. The method of claim 1, wherein processing the first encoded representation with the discrete encoder comprises: converting the first encoded representation to a second encoded representation with the discrete encoder; quantizing the second encoded representation to the first set of discrete features based on a first portion of a target codebook associated with the discrete encoder; and quantizing the second encoded representation to the second set of discrete features based on a second portion of the target codebook associated with the discrete encoder.

3. The method of claim 1, wherein obtaining a first encoded representation associated with training music content comprises: decomposing the training music content into first audio content corresponding to the first music data and second audio content corresponding to the second music data; encoding the first audio content with a first audio encoder to generate a first intermediate encoded representation; encoding the second audio content with a second audio encoder to generate a second intermediate encoded representation; and determining the first encoded representation based on the first intermediate encoded representation and the second intermediate encoded representation.

4. The method of claim 1, wherein the music compression system further comprises a third discrete decoder, the method further comprising: constructing a third set of discrete features based on the first set of discrete features and the second set of discrete features; and decoding the third set of discrete features with the third discrete decoder to generate third audio features.

5. The method of claim 4, wherein the training loss is determined further based on the third audio features.

6. The method of claim 1 or 5, wherein the training loss comprises at least one of: a pitch reconstruction loss, a perceptual reconstruction loss, an adversarial reconstruction loss.

7. The method of claim 1, wherein the first encoded representation is generated by an audio encoder, the audio encoder being a convolutional model.

8. The method of claim 1, wherein the first music data is vocal data and the second music data is accompaniment data.

9. The method of claim 1, further comprising: processing target music content with the trained audio compression system to generate a set of audio tokens. ​ ​ ​ ​ 10. An apparatus for training a music compression system, the music compression system comprising a discrete encoder, a first discrete decoder, and a second discrete decoder, the apparatus comprising: an obtaining module configured to obtain a first encoded representation associated with training music content; an encoding module configured to process the first encoded representation with the discrete encoder to generate a first set of discrete features corresponding to first music data and a second set of discrete features corresponding to second music data; a decoding module configured to decode the first set of discrete features with the first discrete decoder to obtain first audio features corresponding to the first music data and decode the second set of discrete features with the second discrete decoder to obtain second audio features corresponding to the second music data; and a determining module configured to determine a training loss based on the first audio features, the second audio features, and the training music content to adjust parameters of the discrete encoder and the discrete decoder based on the training loss.

11. An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1-9.

12. A computer-readable storage medium having stored thereon a computer program, the computer program being executable by a processor to implement the method according to any one of claims 1-9. ​ ​

Images