Audio track extraction method, apparatus, medium, and computing device

Abstract

Embodiments of the present disclosure provide an audio track extraction method, device, medium and computing device, the method comprising: obtaining a first audio to be extracted and a text command, and determining a first feature vector of the first audio, the text command being used to indicate a first type of audio track to be extracted from the first audio; based on a set time step, obtaining an audio noise at each step of the time step, and generating a noise vector according to each audio noise; de-noising the noise vector according to the first feature vector, the time step and the text command to obtain a target feature vector; and obtaining a second audio according to the target feature vector, the second audio being composed of an audio track corresponding to the first type in the first audio. Through the above method, efficient and accurate separation of a specific audio track in the audio can be achieved.

Description

Technical Field

[0001] The embodiments of this disclosure relate to the field of game technology, and more specifically, the embodiments of this disclosure relate to an audio track extraction method, apparatus, medium, and computing device. Background Technology

[0002] This section is intended to provide background or context for embodiments of this disclosure. The description herein is not intended to imply that it is prior art simply because it is included in this section.

[0003] Audio track splitting refers to the different types of audio tracks resulting from separating the source audio. For example, it involves separating the sound sources of different instruments (or other sound components, such as human voices) in the audio, and the resulting audio tracks contain multiple instruments.

[0004] Current audio source separation solutions typically separate audio sources for specific sound components. If it is necessary to simultaneously obtain different audio tracks from an audio source, multiple audio source separation processes are required, resulting in high costs and low efficiency. Furthermore, current audio source separation solutions generally do not consider situations where the source audio quality is low or damaged, which can easily lead to inaccurate audio separation. Summary of the Invention

[0005] This disclosure provides an audio track extraction method, apparatus, medium, and computing device to at least solve one of the above-mentioned problems.

[0006] In a first aspect of this disclosure, an audio track extraction method is provided, comprising: acquiring a first audio track to be extracted and a text command, and determining a first feature vector of the first audio, wherein the text command is used to indicate a first type of audio track extracted from the first audio; acquiring audio noise at each step of the set time step, and generating a noise vector based on each audio noise; denoising the noise vector based on the first feature vector, the time step, and the text command to obtain a target feature vector; and acquiring a second audio based on the target feature vector, wherein the second audio is composed of audio tracks corresponding to the first type in the first audio.

[0007] In one embodiment of this disclosure, determining the first feature vector of the first audio includes: controlling an encoder in a set model to encode the first audio into a latent vector in the latent space; determining the latent vector as the first feature vector; and obtaining the second audio based on the target feature vector includes: decoding the target feature vector based on a decoder in the set model to obtain the second audio.

[0008] In another embodiment of this disclosure, the training steps of the preset model are as follows: obtaining multiple training samples, the training samples including a third audio and the labels of each type of audio track in the third audio; training the preset model according to each of the training samples to obtain the preset model.

[0009] In yet another embodiment of this disclosure, the loss function of the preset model is updated based on the reconstruction loss function, the adversarial loss function, the adversarial feature loss function, the divergence loss function, and the third audio.

[0010] In another embodiment of this disclosure, obtaining multiple training samples includes: obtaining multiple fourth audio tracks and audio track extraction commands; based on the audio track extraction commands, extracting each second type of audio track specified by the audio track extraction commands from at least one fourth audio track; constructing a fifth audio track corresponding to the audio track extraction commands according to each second type of audio track; determining the tag matching the second type of the audio track in the fifth audio track from each preset tag, and using it as the tag corresponding to the fifth audio track, thereby obtaining the training samples.

[0011] In another embodiment of this disclosure, the step of extracting audio tracks of various second types specified by the audio track extraction command from at least one fourth audio source based on the audio track extraction command includes: determining each second type according to the text information in the audio track extraction command; and extracting the audio track corresponding to each second type from at least one fourth audio source.

[0012] In another embodiment of this disclosure, determining each of the second types based on the text information in the audio track extraction command includes: determining the semantic information of the audio track extraction command; in response to the semantic information indicating that the audio track extraction command is a forward extraction command, determining the types of audio tracks contained in the text information of the audio track extraction command as the second type; and / or in response to the semantic information indicating that the audio track extraction command is a reverse extraction command, determining the types of audio tracks not contained in the text information of the audio track extraction command as the second type.

[0013] In another embodiment of this disclosure, the acquisition of multiple fourth audios includes: acquiring fourth audios from a network, synthesizing fourth audios based on a timbre synthesis method, and / or generating fourth audios based on an audio generation model.

[0014] In another embodiment of this disclosure, each of the preset tags includes multiple levels of tags of the same type, and the tag of the previous level serves as the parent tag of the tag of the next level.

[0015] In another embodiment of this disclosure, the step of denoising the noise vector based on the first feature vector, the time step, and the text command includes: determining a second feature vector corresponding to the time step and a third feature vector corresponding to the text command; and performing denoising on the noise vector for a target number of times based on the first feature vector, the second feature vector, and the third feature vector to obtain a target feature vector, wherein the target number of times is the same as the number of audio noises.

[0016] In another embodiment of this disclosure, denoising the noise vector for a target number of times based on the first feature vector, the second feature vector, and the third feature vector includes: obtaining the first noise vector after the previous denoising, and determining an intermediate vector based on the first feature vector and the first noise feature vector; inputting the intermediate vector, the second feature vector, and the third feature vector to an adapter module; inputting the output parameters of the adapter module, the first noise vector, the second feature vector, and the third feature vector to an audio generation module to obtain a second noise vector output by the audio generation module, and updating the current denoising count; in response to the denoising count being less than the target count, updating the second noise vector to the first noise vector, and returning to the step of determining the intermediate vector based on the first feature vector and the second feature vector.

[0017] In another embodiment of this disclosure, determining the intermediate vector based on the first feature vector and the first noise feature vector includes: performing a convolution operation on the first feature vector with initial zeros to obtain a fourth feature vector; and adding the fourth feature vector to the first noise vector to obtain the intermediate vector.

[0018] In another embodiment of this disclosure, the adapter module includes a plurality of control units connected in sequence and a convolutional layer initially set to zero for each control unit; the audio generation module includes a plurality of encoder units connected in sequence and a decoder unit corresponding to each encoder unit, wherein the decoder units are connected in sequence; wherein, the output parameters of one control unit are input to the next control unit, and the output parameters of the one control unit are input to the convolutional layer corresponding to the one control unit; the output parameters of the convolutional layer are input to the decoder unit corresponding to the convolutional layer; the output parameters of one encoder unit are input to the next encoder unit, and the output parameters of the one encoder unit are input to the decoder unit corresponding to the one encoder unit; the output parameters of one decoder unit are input to the next decoder unit.

[0019] According to a second aspect of the present disclosure, an audio track extraction apparatus is provided, comprising: a first acquisition module, configured to acquire a first audio track to be extracted and a text command, and determine a first feature vector of the first audio track, wherein the text command indicates a first type of audio track extracted from the first audio track; a second acquisition module, configured to acquire audio noise at each step of the set time step, and generate a noise vector based on each audio noise; a denoising module, configured to denoise the noise vector based on the first feature vector, the time step, and the text command to obtain a target feature vector; and a third acquisition module, configured to acquire a second audio track based on the target feature vector, wherein the second audio track is composed of audio tracks corresponding to the first type in the first audio track.

[0020] In one embodiment of this disclosure, the first acquisition module includes: a control unit, configured to control the encoder in the set model to encode the first audio into a latent vector in the latent space; and a determination unit, configured to determine the latent vector as the first feature vector; the third acquisition module includes: a decoding unit, configured to decode the target feature vector based on the decoder in the set model to obtain the second audio.

[0021] In another embodiment of this disclosure, the training module of the preset model includes: a sample acquisition unit for acquiring multiple training samples, the training samples including a third audio and labels for each type of audio track in the third audio; and a training unit for training the preset model according to each of the training samples to obtain the preset model.

[0022] In yet another embodiment of this disclosure, the loss function of the preset model is updated based on the reconstruction loss function, the adversarial loss function, the adversarial feature loss function, the divergence loss function, and the third audio.

[0023] In another embodiment of this disclosure, the sample acquisition unit is specifically configured to: acquire multiple fourth audio tracks and audio track extraction commands; based on the audio track extraction commands, extract each second type of audio track specified by the audio track extraction commands from at least one fourth audio track; construct a fifth audio track corresponding to the audio track extraction commands according to each second type of audio track; determine the tag matching the second type of the audio track in the fifth audio track from each preset tag, and use it as the tag corresponding to the fifth audio track to obtain the training sample.

[0024] In another embodiment of this disclosure, the step of extracting audio tracks of various second types specified by the audio track extraction command from at least one fourth audio source based on the audio track extraction command specifically involves: determining each second type according to the text information in the audio track extraction command; and extracting the audio track corresponding to each second type from at least one fourth audio source.

[0025] In another embodiment of this disclosure, determining each of the second types based on the text information in the audio track extraction command specifically involves: determining the semantic information of the audio track extraction command; in response to the semantic information indicating that the audio track extraction command is a forward extraction command, determining the types of audio tracks contained in the text information of the audio track extraction command as the second type; and / or in response to the semantic information indicating that the audio track extraction command is a reverse extraction command, determining the types of audio tracks not contained in the text information of the audio track extraction command as the second type.

[0026] In another embodiment of this disclosure, obtaining multiple fourth audio files specifically involves: obtaining fourth audio files from a network, synthesizing fourth audio files based on a timbre synthesis method, and / or generating fourth audio files based on an audio generation model.

[0027] In another embodiment of this disclosure, each of the preset tags includes multiple levels of tags of the same type, and the tag of the previous level serves as the parent tag of the tag of the next level.

[0028] In another embodiment of this disclosure, the denoising module includes: a vector determination unit, configured to determine a second feature vector corresponding to the time step and a third feature vector corresponding to the text command; and a multiple denoising unit, configured to denoise the noise vector a target number of times based on the first feature vector, the second feature vector, and the third feature vector to obtain a target feature vector, wherein the target number of times is the same as the number of audio noises.

[0029] In another embodiment of this disclosure, the multiple denoising unit is specifically configured to: obtain a first noise vector after the previous denoising, and determine an intermediate vector based on the first feature vector and the first noise feature vector; input the intermediate vector, the second feature vector, and the third feature vector to the adapter module; input the output parameters of the adapter module, the first noise vector, the second feature vector, and the third feature vector to the audio generation module to obtain a second noise vector output by the audio generation module, and update the current number of denoising iterations; and / or in response to the number of denoising iterations being less than the target number, update the second noise vector to the first noise vector, and return to the step of determining the intermediate vector based on the first feature vector and the second feature vector.

[0030] In another embodiment of this disclosure, determining the intermediate vector based on the first feature vector and the first noise feature vector specifically involves: performing a convolution operation on the first feature vector with initial zeros to obtain a fourth feature vector; and adding the fourth feature vector to the first noise vector to obtain the intermediate vector.

[0031] In another embodiment of this disclosure, the adapter module includes a plurality of control units connected in sequence and a convolutional layer initially set to zero for each control unit; the audio generation module includes a plurality of encoder units connected in sequence and a decoder unit corresponding to each encoder unit, wherein the decoder units are connected in sequence; wherein, the output parameters of one control unit are input to the next control unit, and the output parameters of the one control unit are input to the convolutional layer corresponding to the one control unit; the output parameters of the convolutional layer are input to the decoder unit corresponding to the convolutional layer; the output parameters of one encoder unit are input to the next encoder unit, and the output parameters of the one encoder unit are input to the decoder unit corresponding to the one encoder unit; the output parameters of one decoder unit are input to the next decoder unit.

[0032] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the audio track extraction method provided in any of the first aspects above is implemented.

[0033] According to a fourth aspect of the present disclosure, a computing device is provided, the computing device comprising: at least one processor; and a memory communicatively connected to the at least one processor;

[0034] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the computing device to perform the audio track extraction method provided in any of the first aspects above.

[0035] According to the audio track extraction method, apparatus, medium, and computing device of this disclosure, a first audio track to be extracted and a text command are acquired, and a first feature vector of the first audio track is determined. The text command indicates a first type of audio track extracted from the first audio track. Based on a set time step, audio noise at each step of the time step is acquired, and a noise vector is generated based on each audio noise. The noise vector is then denoised based on the first feature vector, the time step, and the text command to obtain a target feature vector. A second audio track is then acquired based on the target feature vector, and the second audio track is composed of the audio track corresponding to the first type in the first audio track. In this process, the feature vector of the source audio and the text command indicating the audio track type are used to perform audio track extraction processing on the source audio, thereby achieving customized separation and combination extraction of various instruments and vocals in the audio, achieving the technical effect of cost reduction and efficiency improvement. Furthermore, considering the case of low or damaged source audio quality, denoising is performed during the audio track extraction process, which can effectively improve the accuracy of audio separation. Attached Figure Description

[0036] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0037] Figure 1 A schematic diagram illustrating an application scenario of the audio track extraction method according to embodiments of the present disclosure is shown.

[0038] Figure 2 The schematic diagram illustrates a flowchart of an audio track extraction method provided in an embodiment of this disclosure;

[0039] Figure 3 schematically shown Figure 2 A flowchart illustrating step S203;

[0040] Figure 4 The schematic diagram illustrates the structure of the text encoding module in one embodiment of this disclosure;

[0041] Figure 5 A schematic diagram of the structure of an audio generation module in one embodiment of the present disclosure is shown.

[0042] Figure 6A schematic diagram of the adapter module in one embodiment of the present disclosure is shown.

[0043] Figure 7a One of the schematic flowcharts of another audio track splitting method provided in an embodiment of this disclosure is shown;

[0044] Figure 7b The second schematic diagram illustrates another audio track splitting method provided in an embodiment of this disclosure;

[0045] Figure 8 A schematic diagram of the structure of a model in one embodiment of this disclosure is shown.

[0046] Figure 9a A schematic diagram of one embodiment of the label structure is shown.

[0047] Figure 9b A second schematic diagram of a label structure in one embodiment of this disclosure is shown;

[0048] Figure 9c A schematic diagram illustrating a forward extraction method in one embodiment of this disclosure is shown.

[0049] Figure 9d A schematic diagram illustrating a reverse extraction method in one embodiment of this disclosure is shown.

[0050] Figure 10 A schematic flowchart of an audio track splitting method provided in another embodiment of this disclosure is shown.

[0051] Figure 11 A schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present disclosure is shown.

[0052] Figure 12 A schematic diagram of an audio track splitting device according to an embodiment of the present disclosure is shown.

[0053] Figure 13 A schematic diagram of the structure of a computing device provided in an embodiment of the present disclosure is shown.

[0054] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0055] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0056] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0057] According to embodiments of this disclosure, an audio track extraction method, apparatus, medium, and computing device are proposed.

[0058] In this article, it is important to understand that the terms used have the following meanings:

[0059] Diffusion model: A deep learning model architecture that learns the latent structure of a dataset by modeling how data points diffuse in the latent space, thereby achieving the goal of generating target data samples from noise. It is widely used in image generation and audio generation tasks.

[0060] Variational Autoencoder (VAE): Consists of two parts: an encoder and a decoder. The encoder compresses the input data into a latent vector, and the decoder generates new data from this latent vector.

[0061] Latent vectors: Vectors that are compressed and reduced in dimensionality by the VAE encoder to facilitate calculation and reduce computational load. They can be restored to specific content, such as images and audio, by the VAE decoder.

[0062] Audio source separation: Analyze and extract audio from an audio work, and output separate audio tracks for different instruments (or other different audio components), such as guitar, electric guitar, violin, etc.

[0063] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0064] Furthermore, the number of any elements in the accompanying drawings is for illustrative purposes only and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0065] In addition, the data involved in this disclosure may be data authorized by the user or fully authorized by all parties. The collection, dissemination and use of the data shall comply with the requirements of relevant laws and regulations. The implementation methods / executives of this disclosure may be combined with each other.

[0066] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments. Invention Overview

[0068] The inventors have discovered that in audio processing scenarios such as music production and post-processing, music information retrieval, and music copyright protection and management, it is necessary to separate audio sources to obtain different types of music tracks, and then use these tracks for related music production, retrieval, and comparative analysis. Therefore, the accuracy of audio track separation plays a crucial role in the aforementioned audio processing.

[0069] In related technologies, sound source separation techniques for extracting audio tracks are mainly based on convolutional neural networks. For example, one approach involves first extracting human voices from mixed audio using a human voice extraction model to obtain intermediate audio, which includes both human and non-human voice audio frames. Then, a human voice filtering model is used to filter out the non-human voice audio frames in the intermediate audio, resulting in the final human voice audio. Alternatively, one approach involves first performing a discrete Fourier transform on the mixed audio sampling sequence to obtain a frequency domain signal. The modulus of the frequency domain signal is then taken to obtain an initial spectrogram, and the phase of the frequency domain signal is taken to obtain an initial phase map. This initial spectrogram is then input into a convolutional neural network to obtain a human voice mask. The human voice mask is then multiplied by the original spectrogram to obtain a human voice spectrogram. Finally, the human voice spectrogram and the initial phase map are used as the frequency domain signal of the human voice, and an inverse Fourier transform is performed to obtain the time-domain human voice signal, i.e., the separation result. Another approach involves performing sound source separation on the time-domain audio, such as using multi-scale time-domain audio operation methods and temporal convolution. The Transmission Network (TCN) can avoid the phase problem that frequency domain processing methods ignore. The aforementioned sound source separation schemes mainly rely on deep learning models. These models are often limited to separating specific music tracks, such as vocal tracks or guitar tracks, during definition and training. This fixed training method leads to limitations in model functionality; each model can only separate specific instruments or sound components. If different instruments or sound components need to be combined and customized for separation, a separate model needs to be trained for each instrument or sound component, which undoubtedly increases development and usage costs. Furthermore, training each track-separation model independently is not conducive to improving performance. Moreover, the aforementioned sound source separation schemes rely too heavily on specialized track-separation training datasets. These datasets typically contain track-separation information for different instruments or sound components to guide the model training process. However, obtaining track-separation datasets is extremely difficult, requiring not only professional recording equipment and environments but also meticulous editing and processing of the audio signals. Therefore, available track-separation training datasets are relatively few, making it difficult to meet the needs of large-scale training. This scarcity of datasets also results in the model lacking rich prior musical knowledge during training, thus limiting its separation effect and generalization ability.

[0070] Beyond the aforementioned convolutional network-based sound source separation techniques, in the field of artificial intelligence (AI) music generation, with the rapid development of diffusion models in the image domain, AI music generation technologies based on diffusion models are constantly being proposed and upgraded. For example, large-scale music generation models based on text descriptions can generate high-sampling-rate, high-quality music works. After fine-tuning and training the AI ​​text-to-image tool (Stable Diffusion) for image generation, the diffusion model allows the input text description to obtain the spectral information of the music. Another example is the audio generation technology (AudioLDM) based on the latent space diffusion model. After generating the spectrum of music audio from text, a high-fidelity generative adversarial network (HiFi-GAN) is used to complete the spectrum-to-audio conversion. Finally, the AI ​​audio generation model (StableAudio), also based on a diffusion model and operating in the latent space, achieves high-sampling-rate stereo music generation of arbitrary duration by introducing time coding. The aforementioned text-guided music generation technology can currently generate high-quality music audio. However, the current generation chain can only guarantee that the generated music audio is consistent with the input text in terms of macro-semantics. It cannot meet the purpose of extracting specific tracks from the source audio in the audio source separation task. Moreover, the separated instrument tracks should be highly consistent with the corresponding cost in the source audio, which is something that current large-scale music generation models cannot achieve.

[0071] In view of this, the technical solution provided by this disclosure obtains a first audio track to be extracted and a text command, and determines a first feature vector of the first audio track. The text command indicates a first type of audio track extracted from the first audio track. Based on a set time step, the audio noise at each step of the time step is obtained, and a noise vector is generated according to each audio noise. The noise vector is then denoised according to the first feature vector, the time step, and the text command to obtain a target feature vector. Based on the target feature vector, a second audio track is obtained, which is composed of the audio tracks corresponding to the first type in the first audio track. In this process, the feature vector of the source audio and the text command indicating the audio track type are used to perform audio track extraction processing on the source audio, thereby realizing customized separation and combination extraction of various instruments and vocals in the audio, achieving the technical effect of cost reduction and efficiency improvement. Furthermore, considering the case of low or damaged source audio quality, denoising is performed during the audio track extraction process, which can effectively improve the accuracy of audio separation.

[0072] After introducing the basic principles of this disclosure, various non-limiting embodiments of this disclosure will be described in detail below.

[0073] Application Scenarios Overview

[0074] First refer to Figure 1 Examples of application scenarios for the solutions provided in this disclosure are given. Figure 1 This is a schematic diagram illustrating a possible application scenario provided by an embodiment of this disclosure, such as... Figure 1 As shown, in this application scenario, when a user opens an audio program, such as a music application, on terminal device 101, the user can initiate an audio track extraction request to server 102 via terminal device 101 as needed. This request can carry the first audio track to be extracted. After receiving the request, server 102 performs audio track extraction on the first audio track to obtain the second audio track, and feeds the second audio track back to terminal device 101. Terminal device 101 processes the second audio track in the music application, such as audio production or post-processing. Optionally, during the audio track extraction process, server 102 can obtain the denoised target feature vector based on the feature vector of the first audio track and a text command (indicating the first type of audio track extracted from the first audio track) to obtain the extracted second audio track. Optionally, during the audio track extraction process, server 102 can also obtain corresponding models (such as a set model, variational autoencoder, etc.) from server 103 and use the corresponding models to perform audio track extraction. These models are trained in server 103. For example, the music prior knowledge of a large music generation model can be used to train the set model.

[0075] Terminal devices may include, but are not limited to, computers, smartphones, tablets, e-book readers, Moving Picture Experts Group audio layer III (MP3) players, Moving Picture Experts Group audio layer IV (MP4) players, portable computers, in-vehicle computers, wearable devices, desktop computers, set-top boxes, smart TVs, etc.

[0076] In this application scenario, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0077] It should be noted that, Figure 1 This is merely a schematic diagram illustrating one application scenario provided by an embodiment of this disclosure; this embodiment does not necessarily represent an application scenario. Figure 1 The included equipment is not limited, nor is it restricted. Figure 1 The positional relationships between devices are defined. For example, in Figure 1 The application scenario shown may also include a data storage device, which may be an external storage device relative to the server 102 or an internal storage device integrated into the server 102.

[0078] Exemplary methods

[0079] The following is combined Figure 1 Application scenarios, refer to Figure 1 This document describes an audio restoration method according to exemplary embodiments of the present disclosure. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in any way. Rather, the embodiments of the present disclosure can be applied to any applicable scenario.

[0080] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating an audio track extraction method according to an embodiment of the present disclosure. The execution entity of this method can be a server 102, such as... Figure 2 As shown, the method may include steps S201-204:

[0081] Step S201: Obtain the first audio track to be extracted and the text command, and determine the first feature vector of the first audio track. The text command is used to indicate the first type of audio track extracted from the first audio track.

[0082] For example, the first audio track to be extracted and text commands can be obtained from the user terminal. The user can select the first audio track to be extracted according to actual needs on the user terminal. The first audio track and text commands can be obtained from the graphical user interface (GUI) options, such as selecting predefined options in the GUI. For example, in an audio processing software, the user can click a button or select a menu item to select the audio track to be extracted and the audio type, such as "voice," "instrument," or "background noise" for audio track a. Alternatively, they can be obtained through command execution parameters. In the command-line interface, the user can specify the audio type to be extracted through parameters. Alternatively, they can be obtained through a configuration file, i.e., the user can specify the audio type to be extracted by editing the configuration file, which can be JSON, XML, or other formats. In some examples, the audio track extraction method can also be executed by the user terminal, where text commands can be obtained from natural language commands issued by the user. For example, the user can input commands in natural language such as "extract the vocal part from the audio" or "extract background music from the audio," etc.

[0083] As a further example, when a user needs to extract multiple types of audio tracks from a first audio file, they can use text commands to instruct the extraction of various audio track types from the first audio file. For instance, a text command might carry a first type indicating the audio tracks to be extracted from the first audio file, which could include multiple types required by the user (i.e., the first type could include multiple types of audio tracks that the user needs to extract). Alternatively, there could be multiple text commands, each instructing the extraction of one type of audio track (i.e., the first type corresponds to one type). In practical applications, users can make adaptive adjustments according to their actual needs.

[0084] After acquiring the first audio track, to facilitate audio track extraction and noise reduction, a feature vector is determined for the first audio track. This feature vector can also refer to a multi-dimensional feature vector (i.e., a feature vector without dimensionality reduction). The process of obtaining the first feature vector can involve extracting time-domain and frequency-domain features from the first audio track, including time-domain features, frequency-domain features, and time-frequency-domain features. Specifically, time-domain features of the audio signal can be extracted, such as zero-crossing rate, energy, and short-time energy; or frequency-domain features can be extracted using Fourier Transform (FFT), such as spectrum, spectral centroid, and spectral bandwidth; or time-frequency-domain features can be extracted using Short-Time Fourier Transform (STFT) or Mel-frequency cepstral coefficients (MFCC). These extracted features are then combined into a multi-dimensional vector, which serves as the feature vector for the first audio track. Alternatively, other methods can be used to extract the feature vector and audio description information; for example, a deep learning model can be used to determine the feature vector of the first audio track.

[0085] Step S202: Based on the set time step, obtain the audio noise of each step in the time step, and generate a noise vector according to each audio noise.

[0086] In this embodiment, a time step is the smallest unit of time for audio processing. Each time step represents a discrete point in time or a time period at which corresponding audio features, such as notes, chords, and rhythms, can be generated or predicted. Those skilled in the art can adaptively set the step size of the time step according to practical applications; this embodiment does not impose any particular limitation on this. By dividing the audio into multiple time steps, the audio data can be processed step by step, ensuring that the features within each time period can be accurately extracted and processed.

[0087] In this embodiment, the audio noise at each time step is acquired and these audio noises are combined into a large noise vector to facilitate the denoising process in subsequent steps.

[0088] Step S203: Denoise the noise vector based on the first feature vector, the time step, and the text command to obtain the target feature vector.

[0089] In this embodiment, the noise vector is denoised based on the first feature vector, time step, and text command. At each time step, the feature information of the source audio and the text command used to indicate the audio track division are combined to guide the audio noise to be denoised and obtain a vector representation that retains the original audio features (for each audio track).

[0090] For example, a noise vector (Zt) can be constructed by selecting a time step (t) and input into a set model along with a text command (Ctext). This set model can be trained based on a diffusion model. After t time steps of iteration, the latent vector Z0 of the target audio track can be obtained, which is also the target feature vector.

[0091] Optionally, such as Figure 3 As shown, step S203 above, which denoises the noise vector based on the first feature vector, the time step, and the text command, may include the following steps:

[0092] Step S2031: Determine the second feature vector corresponding to the time step and the third feature vector corresponding to the text command.

[0093] In this embodiment, the time step and text command are used as features independent of the audio features (first feature vector) as guiding vectors for denoising.

[0094] In this embodiment, the text command can be an encoded text vector. For example, a text encoding module is used to encode the text prompt, converting the input text into a text vector containing rich semantic information. The text encoding module can use, for example, Contrastive Language-Audio Pretraining (CLAP) and a Text-to-Text Transfer Transformer (T5). CLAP is specifically trained for multimodal audio and text matching; this embodiment can use a CLAP model such as... Figure 4 As shown, in some embodiments, other text encoders may also be used.

[0095] Step S2032: Based on the first feature vector, the second feature vector, and the third feature vector, the noise vector is denoised by a target number of times to obtain a target feature vector, wherein the target number of times is the same as the number of audio noises.

[0096] In this embodiment, the audio noise can be randomly introduced noise (such as noise introduced during the diffusion process of audio data using a diffusion model) or noise inherent in the audio data itself. Typically, there is one random noise at each time step. This embodiment improves the denoising effect by performing a denoising method on the noise vector with respect to the quantity of audio noise, thereby achieving denoising for audio noise of each audio track type.

[0097] In some embodiments, denoising can be performed using a loop to improve the denoising effect. Specifically, in the above steps, denoising the noise vector for a target number of times based on the first feature vector, the second feature vector, and the third feature vector can be performed as follows: obtain the first noise vector after the previous denoising, and determine an intermediate vector based on the first feature vector and the first noise feature vector; input the intermediate vector, the second feature vector, and the third feature vector to the adapter module; input the output parameters of the adapter module, the first noise vector, the second feature vector, and the third feature vector to the audio generation module to obtain the second noise vector output by the audio generation module, and update the current denoising count; in response to the denoising count being less than the target count, update the second noise vector to the first noise vector, and return to the step of determining the intermediate vector based on the first feature vector and the second feature vector.

[0098] In this embodiment, for each denoising step, an intermediate vector is determined by the denoised noise vector and the first feature vector. This intermediate vector carries the feature vector of the noise after the previous denoising. The intermediate vector, the second feature vector corresponding to the time step, and the third feature vector corresponding to the text command are input into the adapter module. The adapter module performs synchronous inference with the audio generation module and integrates the inference results into the steps of the audio generation module. Through the custom adapter module, detailed content information of the source audio can be extracted to guide audio generation, so as to realize that the audio generation model generates high-quality audio that is consistent with the instrument content of the specified type in the source audio.

[0099] In this embodiment, Zx and Ctext are input together into the adapter module. At each time step, the adapter module generates a guiding vector of the same dimension as the audio generation module, guiding the audio generation module to extract and generate instrument tracks that conform to the text commands from the mixed audio. The audio generation module can be implemented based on a pre-trained large-scale music generation model, such as a text-to-audio model (StableAudio) or other models.

[0100] Furthermore, the intermediate vector determined based on the first feature vector and the first noise feature vector in the above steps can be obtained as follows: perform a convolution operation on the first feature vector with initial zeros to obtain a fourth feature vector; add the fourth feature vector to the first noise vector to obtain the intermediate vector.

[0101] In this example approach, for each intermediate vector input into the adapter, a convolution operation with initial zeros is used to preprocess the feature vector, eliminating some unnecessary features or standardizing the features. An intermediate vector is obtained by combining the fourth feature vector (after initial zeros convolution) with the first noise vector. This intermediate vector contains mixed information of the original features and noise, making it more suitable for subsequent denoising steps without introducing other confounding information.

[0102] The above method allows the audio feature vector and noise vector to be superimposed and then input into the adapter for processing, while preserving the feature information of the source audio and avoiding the introduction of other error terms.

[0103] In some embodiments, the adapter module may include a plurality of control units connected in sequence and a convolutional layer initially set to zero for each control unit; the audio generation module includes a plurality of encoder units connected in sequence and a decoder unit corresponding to each encoder unit, wherein each decoder unit is connected in sequence.

[0104] In this system, the output parameters of one control unit are input to the next control unit, and the output parameters of the first control unit are input to the convolutional layer corresponding to the first control unit.

[0105] The output parameters of the convolutional layer are input to the decoder unit corresponding to the convolutional layer.

[0106] The output parameters of one encoder unit are input to the next encoder unit, and the output parameters of the encoder unit are input to the decoder unit corresponding to the encoder unit.

[0107] The output parameters of one decoder unit are input to the next decoder unit.

[0108] For example, the audio generation module can be implemented based on a large text-based music generation model (such as Diffusion, stableaudio, suno, etc.), which has the ability to generate music by taking a desired text description of the music and noise as input and generating high-quality, complete music. Stableaudio will be used as an example to illustrate the audio generation module.

[0109] like Figure 5As shown, the audio generation module can be implemented using the Diffusion architecture, which consists of a 4-level symmetric downsampling encoder block (EncoderBlock) and an upsampling decoder block (DecoderBlock), with skip connections between the downsampling encoder block and the upsampling decoder block. Each block consists of two convolutional residual layers, followed by a series of self-attention and cross-attention layers. During inference, the text command prompt and time step information t are converted into vectors and input together with noise input Zt into the DiffusionUnet, where Zt is initially a random latent vector calculated based on time step t, and Zt-1 is output by DiffusionUnet at each subsequent time step. After T time steps, the final audio latent vector Z0 is obtained, and it is converted into an audio representation (i.e., the second audio) by the VAE decoder.

[0110] As a further example, to facilitate the audio generation module in generating specified track audio from mixed audio by combining text commands, this embodiment employs an adapter module to guide the audio generation process. The adapter can recognize text commands and content in the mixed audio, and achieve fine-grained control over the generation at the sample point level. Figure 6 As shown, the adapter module can include a control module (ControlBlock) with the same structure as the encoder module in the DiffusionUnet network, and zero convolutions (ZeroConv) with all weights initialized to 0. The latent vector Zx of the source audio is processed by ZeroConv, added to Zt, and then input along with the prompt into four cascaded ControlBlocks. The output of each ControlBlock is processed by ZeroConv, added to the corresponding DiffusionUnet EncoderBlock, and then used as the input to the DecoderBlock, thus embedding the content information of the source audio into the generation chain of the audio generation module. Similar to DiffusionUnet, ControlNet operates at each of the T time steps.

[0111] Through the above technical solution, the combined structure between the audio generation module and the adaptation module can achieve fine-grained control and extraction of audio track features at the sampling point level.

[0112] Continue to refer to Figure 2 Step S204: Obtain the second audio based on the target feature vector. The second audio is composed of audio tracks corresponding to the first type in the first audio.

[0113] For example, after obtaining the target feature vector, deep learning methods, such as variational autoencoders (VAEs) or generative adversarial networks (GANs), can be used to obtain the corresponding audio signal based on the feature vector. In some embodiments, the second audio corresponding to the target feature vector can also be obtained by inverting other methods of extracting the first feature vector from the source audio (first audio), such as short-time Fourier transform (STFT), Mel spectral coefficients (MFCC), etc., to extract the first feature vector. MFCC can be used to inversely convert back to Mel spectrum, and then back to linear spectrum. Alternatively, inverse Fourier transform (such as inverse STFT) can be used to convert the spectral data back to a time-domain signal. This disclosure does not particularly limit the process of obtaining the second audio.

[0114] By utilizing the feature vectors of the source audio and text commands indicating the audio track type, the source audio is processed for audio track extraction. This enables customized separation and combination extraction of various instruments and vocals in the audio, achieving the technical effect of cost reduction and efficiency improvement. Furthermore, considering the case of low or damaged source audio quality, noise reduction is performed during the audio track extraction process, which can effectively improve the accuracy of audio separation.

[0115] Figure 7a This is a flowchart illustrating another audio track splitting method provided in an embodiment of this disclosure. Based on the above embodiment, to improve the generalization ability of the data and optimize the audio track splitting extraction effect, the latent vector of the model's codec is set as the feature vector, and the target feature vector is decoded to obtain the second audio corresponding to the audio track splitting. Specifically, in addition to the above steps S201-S204, the determination of the first feature vector of the first audio in step S201 of this embodiment may include the following steps S2011 and S2012, and step S204 is further divided into step S2041:

[0116] Step S2011: Obtain the first audio track to be extracted and the text command, and control the encoder in the set model to encode the first audio into a latent vector in the latent space.

[0117] In this embodiment, the model can be any neural network model with an encoder and a decoder. By controlling the encoder in the model, the first audio to be repaired is input into the encoder, and in the latent space, the encoder encodes the first audio into a latent vector.

[0118] Understandably, a latent space is a low-dimensional representation space that can be used to capture the underlying structure and features of input data. This embodiment uses latent space encoding of latent vectors in a defined model to achieve compression and abstraction of the original audio.

[0119] Step S2012: Determine the latent vector as the first feature vector.

[0120] In this embodiment, by using the latent vector as the first feature vector of the audio, the low-dimensional feature representation facilitates subsequent audio denoising and audio track splitting.

[0121] Step S2041: Based on the decoder in the set model, decode the target feature vector to obtain the second audio.

[0122] For example, the model is set up using a variational autoencoder as an example for illustration, such as... Figure 8 As shown, it includes an encoder and a decoder. The encoder transforms the audio signal x (i.e., the first audio) into a latent space representation z (i.e., the first feature vector). After extracting and generating the audio tracks in the latent space, the decoder transforms the target feature vector z representation in the latent space back into the audio signal x' (the second audio), which can significantly improve the efficiency of the model.

[0123] It is understood that the encoder and decoder described above can be deep learning models, and their inputs and outputs, depending on the implementation, can be time-domain audio signals or the spectrum of audio after Fourier transform. In some embodiments, the encoder and decoder can be components of a large music generation model, whose parameters are trained in the large music generation model and used in conjunction with the music generation module.

[0124] Optionally, this embodiment also performs model training on the server side, specifically, as follows: Figure 7b As shown, the training steps for setting the model are as follows: steps S801 and S802.

[0125] Step S801: Obtain multiple training samples, the training samples including a third audio and the labels of each type of audio track in the third audio.

[0126] In one example, audio data can be collected from at least one audio data source to obtain training samples. For example, multiple training samples can be obtained from online audio libraries, recording devices (using high-quality and low-quality recording devices to record audio separately), or existing datasets (using existing labeled audio datasets).

[0127] In another example, to improve the accuracy of audio track extraction, multiple training samples can be obtained as follows: multiple fourth audio tracks and audio track extraction commands are obtained; based on the audio track extraction commands, each second-type audio track specified by the audio track extraction commands is extracted from at least one fourth audio track; a fifth audio track corresponding to the audio track extraction commands is constructed according to each second-type audio track; from each preset tag, the tag matching the second type of the audio track in the fifth audio track is determined as the tag corresponding to the fifth audio track, thus obtaining the training samples.

[0128] It should be noted that the first type and the second type in this embodiment can refer to the same type or different types. Similarly, the first audio, the second audio, ... can be the same audio or different audio, and this embodiment does not impose any particular limitation on this.

[0129] In this example, each of the preset tags includes multiple levels of tags of the same type, and the tag of the previous level serves as the parent tag of the tag of the next level.

[0130] For example, each audio track can have multiple tags, such as a first-level accompaniment tag, a second-level guitar tag, a stringed instrument tag, a third-level acoustic guitar tag, a bass tag, etc. The first-level accompaniment tag is the parent tag of the second-level guitar tag, and the second-level accompaniment tag is the parent tag of the second-level guitar tag. As a further example, the tag structure can be as follows: Figure 9a , Figure 9b As shown, an instrument tag structure is defined to provide the available instrument selection in text commands. The first level is divided into vocals and accompaniment. The second level of vocals includes lead vocals, harmonies, male vocals, female vocals, and children's vocals. The second level of accompaniment includes percussion, bowed string instruments, plucked string instruments, keyboards, brass instruments, woodwind instruments, and guitar. The third level of accompaniment provides a more refined instrument classification; for example, percussion can be further divided into timpani, snare drum, triangle, etc. These tags are not fixed and can be expanded. Furthermore, tags are not necessarily mutually exclusive; for example, lead vocals can be either male or female, and an acoustic guitar can belong to either the guitar category in the second level or the plucked string instrument category. Next, instrument audio data is collected and labeled. Each audio file can have multiple tags, such as a first-level accompaniment tag, a second-level guitar tag, a second-level plucked string instrument tag, and a third-level acoustic guitar tag, a third-level bass tag, etc.

[0131] To obtain as much sample data as possible while reducing the difficulty of standardization, this embodiment can obtain data (i.e., the fourth audio) in the following ways: by obtaining the fourth audio from the network, synthesizing the fourth audio based on a timbre synthesis method, generating the fourth audio based on an audio generation model, or generating the fourth audio based on both a timbre synthesis method and an audio generation model.

[0132] Specifically, data with instrument tags can be obtained based on existing music libraries or public music datasets, and the instrument tags can be matched with the tag system of this embodiment (e.g., by establishing a mapping relationship); or, the audio of a specified instrument can be synthesized based on the Musical Instrument Digital Interface (MIDI) plus a timbre library synthesis method; or, pure music audio of a specified instrument can be generated based on various current music generation models, and so on.

[0133] In some embodiments, based on the audio track extraction command, extracting each second type of audio track specified by the audio track extraction command from at least one fourth audio can be done in the following manner: determining each second type according to the text information in the audio track extraction command; and extracting the audio track corresponding to each second type from at least one fourth audio.

[0134] In this example, the audio track extraction command can include descriptive text information that can be used to identify and classify the type of audio tracks. For example, the command could include terms like "voice," "background music," or "ambient sound," which help the system determine the second type of each audio track. The fourth audio file can be a composite audio file containing multiple audio tracks. Based on the second type determined in the previous step, the system can find and extract the corresponding audio tracks within this composite audio file. For example, if the second type is "voice," the system will find and extract the audio track containing voices in the fourth audio file; if the second type is "background music," the system will extract the audio track containing background music.

[0135] In this way, the text information in the audio track extraction command can be extracted to accurately identify and extract the required audio tracks, thereby enabling the classification and processing of audio content.

[0136] Furthermore, forward and reverse extraction methods can be used to determine the audio track type to improve the efficiency of training sample acquisition. Specifically, the determination of each second type based on the text information in the audio track extraction command can be achieved as follows: determine the semantic information of the audio track extraction command; in response to the semantic information indicating that the audio track extraction command is a forward extraction command, determine the audio track type contained in the text information of the audio track extraction command as the second type; and / or, in response to the semantic information indicating that the audio track extraction command is a reverse extraction command, determine the audio track type not contained in the text information of the audio track extraction command as the second type.

[0137] For example, after constructing the audio dataset, text commands and corresponding audio data can be dynamically and randomly constructed during training based on a pre-defined text template (which can be adaptively determined by those skilled in the art in conjunction with actual applications). Specifically, combined with Figure 9c and Figure 9d As shown, it can include the following methods:

[0138] Forward instrument extraction: For example, to "extract vocals and guitar from audio," the data construction rule is as follows: Extract an audio file V with only vocals as the primary label and an audio file J with only guitar as the secondary label from the dataset. Then randomly select i (>=0) audio files, ensuring that the primary label does not contain vocals, the secondary label does not contain guitar, and the same tertiary label is not selected repeatedly. Sum all audio files to obtain the mixed audio file x, sum audio files V and J to obtain the target audio file t, and use the prompt text as the prompt, thus forming a training dataset.

[0139] Reverse instrument extraction: For example, "remove vocals and guitar from audio" or "extract other instruments besides vocals and guitar from audio," the data construction rules are the same as above. Sum all audios to get the mixed audio x, sum all audios except audio V and audio J to get the target t, and use the prompt text as the prompt, thus forming a set of training data.

[0140] In some examples, forward instrument extraction and reverse instrument extraction can be performed simultaneously to form corresponding training samples.

[0141] Continue to refer to Figure 8 Step S802: Train the preset model according to each of the training samples to obtain the set model.

[0142] For example, the loss function of the preset model in this embodiment can be updated based on the reconstruction loss function, the adversarial loss function, the adversarial feature loss function, the divergence loss function, and the third audio.

[0143] Understandably, in machine learning and deep learning, a loss function is a function used to measure the difference between the model's predictions and the actual results.

[0144] In this example, for the preset model, the loss function can be designed and updated based on the following types: a reconstruction loss function, which measures the accuracy of the model in reconstructing input data, and can include mean squared error (MSE) and mean absolute error (MAE). In the audio track extraction task of this embodiment, the reconstruction loss function can be used to evaluate the difference between the audio generated by the model and the original audio; an adversarial loss function, which can be used in generative adversarial networks (GANs) to measure the game effect between the generator and the discriminator, and can employ cross-entropy loss. In the audio track extraction task, the generator attempts to generate realistic audio segments, while the discriminator attempts to distinguish between the generated audio and the real audio. The adversarial loss function helps the generator generate more realistic audio; the adversarial feature loss function is used to measure the difference between the generated features and the real features, and can be used to improve the feature representation of the generative model. For example, feature matching loss can be used. In audio track extraction tasks, this loss function helps the generator produce audio with specific features, such as a certain timbre or frequency characteristic. The divergence loss function measures the difference between two probability distributions, where divergence measures can include Kullback-Leibler divergence (KL divergence) and Jensen-Shannon divergence (JS divergence). In the audio track extraction of this embodiment, the divergence loss function can be used to measure the difference between the generated audio distribution and the real audio distribution, thereby improving the performance of the generative model. Furthermore, the loss function can also be based on updates to a third audio track; in other words, the loss function considers not only the current input and output but also additional audio data. For example, in an audio track intensity conversion task, the third audio track can be a reference audio track of the target type, and the loss function can measure the difference between the generated audio track and the reference audio track. In some embodiments, any one of the above-mentioned loss functions or a combination of the above-mentioned different types of loss functions can be used to design a more complex and effective preset model loss function, thereby improving the model's performance in various audio track extraction tasks.

[0145] Compared to existing audio source separation methods limited by finite track-by-track datasets, which often fail to fully capture the complexity and diversity of music, this disclosure combines a large-scale music generation model with audio source separation technology, creating a novel solution. By accumulating a wealth of prior musical knowledge during the training of the large model—including the recognition of different instrument sounds, understanding of musical structure, and mastery of sound mixing—the model can incorporate this rich prior musical knowledge into the audio source separation process, guiding it to more accurately separate new audio data. This allows the model to accurately identify and separate different audio sources even without a large amount of track-by-track data. Through this combination, an end-to-end audio source separation system can be constructed. This system can not only automatically identify different audio sources in audio but also perform customized separation and combination based on user text input, bringing a more efficient and intelligent solution to the fields of audio processing and music production.

[0146] In one exemplary embodiment, arbitrary target track audio is extracted by inputting a text command from mixed music audio. This can be based on a pre-trained music generation model (i.e., an audio generation model) to fully utilize prior music data, achieving high-quality extraction and identification of different instrument tracks. Pre-defined text prompt templates and data composition methods are used to train the model, adapter module, and other models, enabling them to incorporate text prompts into the audio generation process and guide the model to generate the specified track audio. Specifically, the network architecture of the technical solution provided in this embodiment may include: an audio latent space encoding / decoding module, an audio generation module, a text encoding module, and an adapter module. An exemplary processing flow is as follows: Figure 10 As shown:

[0147] First, the desired track extraction text command `prompt` is input, which can be encoded into a vector `Ctext` using a pre-trained text encoder. The mixed audio `x` is input, and the audio latent space encoder (the setting model) encodes it into a latent space vector `Zx` (i.e., the first feature vector). At a selected time step `t`, a noise vector `Zt` is constructed and input along with `Ctext` into the audio generation module. After t time steps of iteration, the latent vector `Z0` (i.e., the target feature vector) of the target track audio is obtained. Simultaneously, `Zx` and `Ctext` are input into the adapter module. At each time step, the adapter module generates a guide vector of the same dimension as the generation module, guiding the audio generation module to extract the instrument tracks from the mixed audio that conform to the text command. In this example, the text encoder, audio latent space encoder / decoder, and audio generation module are implemented based on a pre-trained music generation model. Their weights can be directly taken from the pre-trained model and fixed to reuse the prior music knowledge in the pre-trained model. The setting model and adapter module can be trained based on a constructed audio source separation dataset.

[0148] In summary, the audio track extraction scheme based on text commands in this disclosure can satisfy the input of text commands to perform customized separation and combination extraction of various instruments and vocals in audio; by utilizing the sound source separation scheme based on the prior music knowledge in a pre-trained music generation model, it can complete the task of extracting high-quality music for a specified instrument even without a detailed music track extraction dataset; and by using a custom adapter module, it can extract detailed content information of the source audio to guide audio generation and ensure that the audio generation model generates high-quality audio consistent with the specified instrument content in the source audio; furthermore, through structured multi-level instrument tags, it can utilize audio from multiple channels to optimize the sound source separation effect, and proposes forward and reverse sound source separation text command methods, providing corresponding training data combination methods, thereby achieving training optimization of the entire audio track extraction process.

[0149] Exemplary media

[0150] After introducing the methods of exemplary embodiments of this disclosure, the following references are made. Figure 11 A computer-readable storage medium for exemplary embodiments of the present disclosure will be described.

[0151] refer to Figure 11 As shown, the storage medium 30 stores a program product for implementing the above-described method according to embodiments of the present disclosure. This program product may be a portable compact disc read-only memory (CD-ROM) and includes program code, and can run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.

[0152] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0153] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium.

[0154] Program code for performing the operations disclosed herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).

[0155] Exemplary device

[0156] Having introduced the medium of exemplary embodiments of this disclosure, the following references are made to... Figure 12 The audio restoration apparatus of the exemplary embodiments of this disclosure is described to implement the method in any of the above method embodiments. The implementation principle and technical effect are similar, and will not be repeated here.

[0157] Figure 12 A schematic diagram of an audio track extraction device according to an embodiment of this disclosure is shown, as follows: Figure 12As shown, the device includes a first acquisition module 121, a second acquisition module 122, a denoising module 123, and a third acquisition module 124. The first acquisition module 121 is used to acquire a first audio track to be extracted and a text command, and to determine a first feature vector of the first audio track. The text command indicates a first type of audio track extracted from the first audio track. The second acquisition module 122 is used to acquire audio noise at each step of a set time step, and to generate a noise vector based on each audio noise. The denoising module 123 is used to denoise the noise vector based on the first feature vector, the time step, and the text command to obtain a target feature vector. The third acquisition module 124 is used to acquire a second audio track based on the target feature vector, the second audio track being composed of audio tracks corresponding to the first type in the first audio track.

[0158] In one embodiment of this disclosure, the first acquisition module 121 includes: a control unit, configured to control the encoder in the set model to encode the first audio into a latent vector in the latent space; and a determination unit, configured to determine the latent vector as the first feature vector; the third acquisition module includes: a decoding unit, configured to decode the target feature vector based on the decoder in the set model to obtain the second audio.

[0159] In another embodiment of this disclosure, the training module of the preset model includes: a sample acquisition unit for acquiring multiple training samples, the training samples including a third audio and labels for each type of audio track in the third audio; and a training unit for training the preset model according to each of the training samples to obtain the preset model.

[0160] In yet another embodiment of this disclosure, the loss function of the preset model is updated based on the reconstruction loss function, the adversarial loss function, the adversarial feature loss function, the divergence loss function, and the third audio.

[0161] In another embodiment of this disclosure, the sample acquisition unit is specifically configured to: acquire multiple fourth audio tracks and audio track extraction commands; based on the audio track extraction commands, extract each second type of audio track specified by the audio track extraction commands from at least one fourth audio track; construct a fifth audio track corresponding to the audio track extraction commands according to each second type of audio track; determine the tag matching the second type of the audio track in the fifth audio track from each preset tag, and use it as the tag corresponding to the fifth audio track to obtain the training sample.

[0162] In another embodiment of this disclosure, the step of extracting audio tracks of various second types specified by the audio track extraction command from at least one fourth audio source based on the audio track extraction command specifically involves: determining each second type according to the text information in the audio track extraction command; and extracting the audio track corresponding to each second type from at least one fourth audio source.

[0163] In another embodiment of this disclosure, determining each of the second types based on the text information in the audio track extraction command specifically involves: determining the semantic information of the audio track extraction command; in response to the semantic information indicating that the audio track extraction command is a forward extraction command, determining the types of audio tracks contained in the text information of the audio track extraction command as the second type; and in response to the semantic information indicating that the audio track extraction command is a reverse extraction command, determining the types of audio tracks not contained in the text information of the audio track extraction command as the second type.

[0164] In another embodiment of this disclosure, obtaining multiple fourth audio files specifically involves: obtaining fourth audio files from a network, synthesizing fourth audio files based on a timbre synthesis method, and / or generating fourth audio files based on an audio generation model.

[0165] In another embodiment of this disclosure, each of the preset tags includes multiple levels of tags of the same type, and the tag of the previous level serves as the parent tag of the tag of the next level.

[0166] In another embodiment of this disclosure, the denoising module 123 includes: a vector determination unit, configured to determine a second feature vector corresponding to the time step and a third feature vector corresponding to the text command; and a multiple denoising unit, configured to perform denoising on the noise vector a target number of times based on the first feature vector, the second feature vector, and the third feature vector to obtain a target feature vector, wherein the target number of times is the same as the number of audio noises.

[0167] In another embodiment of this disclosure, the multiple denoising unit is specifically configured to: obtain a first noise vector after the previous denoising, and determine an intermediate vector based on the first feature vector and the first noise feature vector; input the intermediate vector, the second feature vector, and the third feature vector to an adapter module; input the output parameters of the adapter module, the first noise vector, the second feature vector, and the third feature vector to an audio generation module to obtain a second noise vector output by the audio generation module, and update the current denoising count; in response to the denoising count being less than the target count, update the second noise vector to the first noise vector, and return to the step of determining the intermediate vector based on the first feature vector and the second feature vector.

[0168] In another embodiment of this disclosure, determining the intermediate vector based on the first feature vector and the first noise feature vector specifically involves: performing a convolution operation on the first feature vector with initial zeros to obtain a fourth feature vector; and adding the fourth feature vector to the first noise vector to obtain the intermediate vector.

[0169] In another embodiment of this disclosure, the adapter module includes a plurality of control units connected in sequence and a convolutional layer initially set to zero for each control unit; the audio generation module includes a plurality of encoder units connected in sequence and a decoder unit corresponding to each encoder unit, wherein the decoder units are connected in sequence; wherein, the output parameters of one control unit are input to the next control unit, and the output parameters of the one control unit are input to the convolutional layer corresponding to the one control unit; the output parameters of the convolutional layer are input to the decoder unit corresponding to the convolutional layer; the output parameters of one encoder unit are input to the next encoder unit, and the output parameters of the one encoder unit are input to the decoder unit corresponding to the one encoder unit; the output parameters of one decoder unit are input to the next decoder unit.

[0170] Exemplary computing device

[0171] Having described the methods, media, and apparatus of exemplary embodiments of this disclosure, the following references... Figure 13 A computing device according to an exemplary embodiment of the present disclosure will be described.

[0172] Figure 13 The computing device 40 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein. Figure 13As shown, the computing device 40 is presented in the form of a general-purpose computing device. Components of the computing device 40 may include, but are not limited to: at least one processing unit 401, at least one storage unit 402, and a bus 403 connecting different system components (including the processing unit 401 and the storage unit 402). The at least one storage unit 402 stores computer-executable instructions; the at least one processing unit 401 includes a processor that executes the computer-executable instructions to implement the methods described above. The bus 403 includes a data bus, a control bus, and an address bus. The storage unit 402 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 4021 and / or cache memory 4022, and may further include a readable medium in the form of non-volatile memory, such as read-only memory (ROM) 4023.

[0173] Storage unit 402 may also include a program / utility 4025 having a set (at least one) of program modules 4024, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment. Computing device 40 may also communicate with one or more external devices 404 (e.g., a keyboard, pointing device, etc.). This communication may be performed via input / output (I / O) interface 405. Furthermore, computing device 40 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 406. Figure 13 As shown, network adapter 406 communicates with other modules of computing device 40 via bus 403. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with computing device 40, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0174] It should be noted that although several units / modules or sub-units / modules of the audio restoration apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0175] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0176] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. An audio track extraction method, characterized by, The method comprises: obtaining a first audio to be extracted and a text command, and determining a first feature vector of the first audio, the text command being used to indicate a first type of audio track extracted from the first audio; based on a set time step, obtaining audio noise of each step in the time step, and generating a noise vector according to each audio noise; according to the first feature vector, a second feature vector corresponding to the time step, and a third feature vector corresponding to the text command, denoising the noise vector a target number of times to obtain a target feature vector; wherein the target number is the same as the number of audio noises; obtaining a second audio according to the target feature vector, the second audio being composed of an audio track corresponding to the first type in the first audio.

2. The audio track extraction method of claim 1, wherein, The method comprises: controlling an encoder in a set model to encode the first audio into a hidden vector in a hidden space; determining the hidden vector as the first feature vector; The method comprises: decoding the target feature vector based on a decoder in the set model to obtain the second audio; wherein the set model is used to determine the first feature vector of the first audio, and to decode the target feature vector to obtain the second audio.

3. The audio track extraction method of claim 2, wherein, The training steps of the set model are as follows: obtaining a plurality of training samples, the training samples comprising a third audio and a label of each type of audio track in the third audio, the label being used to provide a supervision constraint for an encoder and a decoder of a preset model; training the preset model according to each training sample to obtain the set model.

4. The audio track extraction method of claim 3, wherein, The loss function of the preset model is updated based on at least one of a reconstruction loss function, an adversarial loss function, an adversarial feature loss function, a divergence loss function, or the third audio.

5. The audio track extraction method of claim 3, wherein, The method comprises: obtaining a plurality of fourth audios and an audio track extraction command; based on the audio track extraction command, extracting each second type of audio track specified by the audio track extraction command from at least one fourth audio; constructing a fifth audio corresponding to the audio track extraction command according to each second type of audio track; determining a label matched with the second type of audio track in the fifth audio from each preset label as a label corresponding to the fifth audio to obtain the training sample.

6. The audio track extraction method of claim 5, wherein, The method comprises: determining each second type according to text information in the audio track extraction command; extracting an audio track corresponding to each second type from at least one fourth audio.

7. The audio track extraction method of claim 6, wherein, The method comprises: determining semantic information of the audio track extraction command; determining, as the second type, a type of the audio track included in the text information in the audio track extraction command, in response to the semantic information indicating that the audio track extraction command is a forward extraction command; and / or determining, as the second type, a type of the audio track not included in the text information in the audio track extraction command, in response to the semantic information indicating that the audio track extraction command is a reverse extraction command.

8. The audio track extraction method of claim 5, wherein, The obtaining of the plurality of fourth audios comprises: obtaining the fourth audios from a network, synthesizing the fourth audios based on a timbre synthesis manner, and / or generating the fourth audios based on an audio generation model.

9. The audio track extraction method of claim 5, wherein, Each of the preset labels includes a plurality of levels of labels of the same category, and a label of a previous level serves as a parent label of a label of a next level.

10. The audio demixing method of any one of claims 1-9, wherein, The method further comprises: determining a second feature vector corresponding to the time step and a third feature vector corresponding to the text command.

11. The audio track extraction method of claim 10, wherein, The de-noising of the noise vector a target number of times according to the first feature vector, the second feature vector corresponding to the time step, and the third feature vector corresponding to the text command comprises: obtaining a first noise vector after the last de-noising, and determining an intermediate vector according to the first feature vector and the first noise vector; inputting the intermediate vector, the second feature vector, and the third feature vector into an adapter module, the adapter module being configured to process the received intermediate vector, second feature vector, and third feature vector, and obtain an output parameter; inputting the output parameter of the adapter module, the first noise vector, the second feature vector, and the third feature vector into an audio generation module, obtaining a second noise vector output by the audio generation module, and updating a current de-noising number; in response to the de-noising number being less than the target number, updating the second noise vector to the first noise vector, and returning to the step of determining the intermediate vector according to the first feature vector and the second feature vector.

12. The audio track extraction method of claim 11, wherein, The determining of the intermediate vector according to the first feature vector and the first noise vector comprises: performing an initial zero convolution operation on the first feature vector to obtain a fourth feature vector; adding the fourth feature vector and the first noise vector to obtain the intermediate vector.

13. The audio track extraction method of claim 11, wherein, The adapter module comprises a plurality of control units connected in sequence and an initial zero convolution layer corresponding to each control unit; the audio generation module comprises a plurality of encoder units connected in sequence and a decoder unit corresponding to each encoder unit, and each decoder unit is connected in sequence; wherein the output parameter of one control unit is input into a next control unit, and the output parameter of the one control unit is input into the convolution layer corresponding to the one control unit; the output parameter of the convolution layer is input into the decoder unit corresponding to the convolution layer; the output parameter of one encoder unit is input into a next encoder unit, and the output parameter of the one encoder unit is input into the decoder unit corresponding to the one encoder unit. The output parameter of one decoder unit is input to the next decoder unit.

14. An audio track extraction apparatus characterized by comprising: Comprise: The first acquisition module is used for acquiring the first audio of the audio track to be extracted and a text command, and determining a first feature vector of the first audio, wherein the text command is used for indicating a first type of audio track extracted from the first audio; The second acquisition module is used for acquiring audio noise of each step in a set time step, and generating a noise vector according to the audio noise; The denoising module is used for performing denoising on the noise vector a target number of times according to the first feature vector, a second feature vector corresponding to the time step, and a third feature vector corresponding to the text command, to obtain a target feature vector; wherein the target number is the same as the number of the audio noise; The third acquisition module is used for acquiring a second audio according to the target feature vector, wherein the second audio is composed of an audio track corresponding to the first type in the first audio.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, the method in any one of claims 1 to 13 is implemented.

16. A computing device, comprising: Comprise: At least one processor; And a memory connected with the at least one processor in communication; Wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the computing device to execute the method in any one of claims 1 to 13.

Images