Audio sub-track extraction method and device, medium and computing equipment
By obtaining the feature vectors and text commands of the audio, and generating noise vectors based on time steps for denoising, realizing custom separation and combination extraction of audio separation, solving the problems of high cost and low accuracy in the prior art, improving the The efficiency and accuracy of audio separation are achieved.
Patent Information
- Application Number
- CN202410997909.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-23
AI Technical Summary
The existing audio source separation technology is costly and inefficient, and it is difficult to deal with low or damaged source audio quality, resulting in inaccurate audio separation.
By obtaining the feature vector and text commands of the audio to be extracted, the noise vector is generated based on the time steps, and the target feature vector is obtained through the denoising process, thereby realizing custom separation and combination extraction of audio track division.
Reduces the cost of audio separation, improves efficiency, and improves the accuracy of audio separation when source audio quality is low or damaged.
Smart Images

Figure CN118942469A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the field of game technology, and more specifically, the embodiments of the present disclosure relate to an audio track extraction method, apparatus, medium and computing device. Background Art
[0002] This section is intended to provide a background or context to the embodiments of the present disclosure. No description herein is admitted to be prior art by virtue of its inclusion in this section.
[0003] Audio tracks refer to different types of track audio after separating the source audio. For example, the sound sources of different instruments (or other sound components, such as human voice, etc.) in the audio are separated, and the separated audio contains audio tracks of multiple instruments.
[0004] Current sound source separation solutions usually separate specific sound components. If different audio tracks in the sound source need to be obtained at the same time, multiple sound source separations are required, resulting in high costs and low efficiency. In addition, current sound source separation solutions usually do not consider situations where the source audio quality is low or damaged, which can easily lead to inaccurate audio separation. Summary of the invention
[0005] The present disclosure provides an audio track extraction method, apparatus, medium and computing device to solve at least one of the above problems.
[0006] In a first aspect of an embodiment of the present disclosure, a method for extracting an audio track is provided, comprising: obtaining a first audio and a text command of an audio track to be extracted, and determining a first feature vector of the first audio, wherein the text command is used to indicate a first type of an audio track extracted from the first audio; based on a set time step, obtaining audio noise at each step in the time step, and generating a noise vector according to each audio noise; denoising the noise vector according to the first feature vector, the time step and the text command to obtain a target feature vector; obtaining a second audio according to the target feature vector, wherein the second audio is composed of audio tracks corresponding to the first type in the first audio.
[0007] In one embodiment of the present disclosure, determining the first feature vector of the first audio includes: controlling an encoder in a set model to encode the first audio into a latent vector in a latent space; determining the latent vector as the first feature vector; and obtaining the second audio based on the target feature vector includes: decoding the target feature vector based on a decoder in the set model to obtain the second audio.
[0008] In another embodiment of the present disclosure, the training steps of the set model are as follows: obtain multiple training samples, the training samples include the third audio and the label of each type of audio track in the third audio; according to each of the training samples, train the preset model to obtain the set model.
[0009] In another embodiment of the present disclosure, the loss function of the preset model is updated based on the reconstruction loss function, the adversarial loss function, the adversarial feature loss function, the divergence loss function, and the third audio.
[0010] In another embodiment of the present disclosure, the obtaining of multiple training samples includes: obtaining multiple fourth audios and audio track extraction commands; based on the audio track extraction command, extracting, from at least one fourth audio, each second type of audio track specified to be extracted by the audio track extraction command; constructing a fifth audio corresponding to the audio track extraction command according to each of the second type of audio tracks; and determining, from each preset label, a label that matches the second type of audio track in the fifth audio as the label corresponding to the fifth audio, to obtain the training sample.
[0011] In yet another embodiment of the present disclosure, the method of extracting, from at least one fourth audio, each second type of audio track specified by the audio track extraction command based on the audio track extraction command comprises: determining each second type based on text information in the audio track extraction command; and extracting, from at least one of the fourth audios, an audio track corresponding to each second type.
[0012] In still another embodiment of the present disclosure, determining each second type based on the text information in the audio track extraction command includes: determining semantic information of the audio track extraction command; in response to the semantic information indicating that the audio track extraction command is a forward extraction command, determining the type of the audio track contained in the text information in the audio track extraction command as the second type; and / or in response to the semantic information indicating that the audio track extraction command is a reverse extraction command, determining the type of the audio track not contained in the text information in the audio track extraction command as the second type.
[0013] In yet another embodiment of the present disclosure, the obtaining of multiple fourth audios includes: obtaining the fourth audio from a network, synthesizing the fourth audio based on a timbre synthesis method, and / or generating the fourth audio based on an audio generation model.
[0014] In yet another embodiment of the present disclosure, each of the preset tags includes tags of multiple levels of the same type, and a tag of an upper level serves as a parent tag of a tag of a lower level.
[0015] In still another embodiment of the present disclosure, the denoising of the noise vector according to the first eigenvector, the time step and the text command includes: determining a second eigenvector corresponding to the time step and a third eigenvector corresponding to the text command; denoising the noise vector a target number of times according to the first eigenvector, the second eigenvector and the third eigenvector to obtain a target eigenvector, wherein the target number is the same as the amount of the audio noise.
[0016] In another embodiment of the present disclosure, the noise vector is denoised a target number of times according to the first eigenvector, the second eigenvector and the third eigenvector, including: obtaining the first noise vector after the last denoising, and determining an intermediate vector according to the first eigenvector and the first noise eigenvector; inputting the intermediate vector, the second eigenvector and the third eigenvector into an adapter module; inputting the output parameters of the adapter module, the first noise vector, the second eigenvector and the third eigenvector into an audio generation module to obtain the second noise vector output by the audio generation module, and updating the current denoising times; in response to the denoising times being less than the target times, updating the second noise vector to the first noise vector, and returning to execute the step of determining the intermediate vector according to the first eigenvector and the second eigenvector.
[0017] In still another embodiment of the present disclosure, determining the intermediate vector based on the first eigenvector and the first noise eigenvector includes: performing a convolution operation with an initial zero setting on the first eigenvector to obtain a fourth eigenvector; and adding the fourth eigenvector and the first noise vector to obtain the intermediate vector.
[0018] In another embodiment of the present disclosure, the adapter module includes a plurality of control units connected in sequence and a convolutional layer initially set to zero corresponding to each of the control units; the audio generation module includes a plurality of encoder units connected in sequence and a decoder unit corresponding to each of the encoder units, and the decoder units are connected in sequence; wherein the output parameters of a control unit are input to the next control unit, and the output parameters of the control unit are input to the convolutional layer corresponding to the control unit; the output parameters of the convolutional layer are input to the decoder unit corresponding to the convolutional layer; the output parameters of an encoder unit are input to the next encoder unit, and the output parameters of the encoder unit are input to the decoder unit corresponding to the encoder unit; the output parameters of a decoder unit are input to the next decoder unit.
[0019] According to a second aspect of an embodiment of the present disclosure, there is provided an audio track extraction device, comprising: a first acquisition module, used to acquire a first audio and a text command of an audio track to be extracted, and determine a first feature vector of the first audio, wherein the text command is used to indicate a first type of an audio track extracted from the first audio; a second acquisition module, used to acquire the audio noise of each step in the time step based on a set time step, and generate a noise vector according to each audio noise; a denoising module, used to denoise the noise vector according to the first feature vector, the time step and the text command, to obtain a target feature vector; and a third acquisition module, used to acquire a second audio according to the target feature vector, wherein the second audio is composed of audio tracks corresponding to the first type in the first audio.
[0020] In one embodiment of the present disclosure, the first acquisition module includes: a control unit, used to control the encoder in the set model to encode the first audio into a latent vector in the latent space; a determination unit, used to determine the latent vector as the first feature vector; the third acquisition module includes: a decoding unit, used to decode the target feature vector based on the decoder in the set model to obtain the second audio.
[0021] In another embodiment of the present disclosure, the training module of the set model includes: a sample acquisition unit, used to acquire multiple training samples, the training samples including a third audio and a label of each type of audio track in the third audio; a training unit, used to train a preset model according to each of the training samples to obtain the set model.
[0022] In another embodiment of the present disclosure, the loss function of the preset model is updated based on the reconstruction loss function, the adversarial loss function, the adversarial feature loss function, the divergence loss function, and the third audio.
[0023] In another embodiment of the present disclosure, the sample acquisition unit is specifically used to: acquire multiple fourth audios and audio track extraction commands; based on the audio track extraction command, extract each second type of audio track specified to be extracted by the audio track extraction command from at least one fourth audio; construct a fifth audio corresponding to the audio track extraction command according to each second type of audio track; determine, from each preset label, a label matched by the second type of audio track in the fifth audio as the label corresponding to the fifth audio, and obtain the training sample.
[0024] In yet another embodiment of the present disclosure, based on the audio track extraction command, extracting each second type of audio track specified by the audio track extraction command from at least one fourth audio, specifically: determining each second type according to text information in the audio track extraction command; and extracting each audio track corresponding to the second type in at least one of the fourth audios.
[0025] In still another embodiment of the present disclosure, the determining of each second type according to the text information in the audio track extraction command is specifically as follows: determining the semantic information of the audio track extraction command; in response to the semantic information indicating that the audio track extraction command is a forward extraction command, determining the type of the audio track contained in the text information in the audio track extraction command as the second type; and / or in response to the semantic information indicating that the audio track extraction command is a reverse extraction command, determining the type of the audio track not contained in the text information in the audio track extraction command as the second type.
[0026] In yet another embodiment of the present disclosure, the obtaining of multiple fourth audios specifically includes: obtaining the fourth audio from a network, synthesizing the fourth audio based on a timbre synthesis method, and / or generating the fourth audio based on an audio generation model.
[0027] In yet another embodiment of the present disclosure, each of the preset tags includes tags of multiple levels of the same type, and a tag of an upper level serves as a parent tag of a tag of a lower level.
[0028] In another embodiment of the present disclosure, the denoising module includes: a vector determination unit, used to determine the second eigenvector corresponding to the time step and the third eigenvector corresponding to the text command; a multiple denoising unit, used to denoise the noise vector a target number of times according to the first eigenvector, the second eigenvector and the third eigenvector to obtain a target eigenvector, and the target number is the same as the amount of audio noise.
[0029] In another embodiment of the present disclosure, according to the multiple denoising units, it is specifically used to: obtain the first noise vector after the last denoising, and determine the intermediate vector according to the first eigenvector and the first noise eigenvector; input the intermediate vector, the second eigenvector and the third eigenvector into the adapter module; input the output parameters of the adapter module, the first noise vector, the second eigenvector and the third eigenvector into the audio generation module to obtain the second noise vector output by the audio generation module, and update the current denoising times; and / or in response to the denoising times being less than the target times, update the second noise vector to the first noise vector, and return to execute the step of determining the intermediate vector according to the first eigenvector and the second eigenvector.
[0030] In still another embodiment of the present disclosure, the determining of the intermediate vector based on the first eigenvector and the first noise eigenvector is specifically as follows: performing a convolution operation with an initial zero setting on the first eigenvector to obtain a fourth eigenvector; and adding the fourth eigenvector and the first noise vector to obtain the intermediate vector.
[0031] In another embodiment of the present disclosure, the adapter module includes a plurality of control units connected in sequence and a convolutional layer initially set to zero corresponding to each of the control units; the audio generation module includes a plurality of encoder units connected in sequence and a decoder unit corresponding to each of the encoder units, and the decoder units are connected in sequence; wherein the output parameters of a control unit are input to the next control unit, and the output parameters of the control unit are input to the convolutional layer corresponding to the control unit; the output parameters of the convolutional layer are input to the decoder unit corresponding to the convolutional layer; the output parameters of an encoder unit are input to the next encoder unit, and the output parameters of the encoder unit are input to the decoder unit corresponding to the encoder unit; the output parameters of a decoder unit are input to the next decoder unit.
[0032] According to a third aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the audio track extraction method provided in any one of the first aspects above is implemented.
[0033] According to a fourth aspect of the embodiments of the present disclosure, there is provided a computing device, the computing device comprising: at least one processor; and a memory communicatively connected to the at least one processor;
[0034] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the computing device to perform the audio track extraction method provided in any one of the first aspects above.
[0035] According to the audio sub-track extraction method, device, medium and computing device of the embodiment of the present disclosure, by obtaining the first audio and text command of the audio sub-track to be extracted, and determining the first feature vector of the first audio, the text command is used to indicate the first type of the audio sub-track extracted from the first audio, based on the set time step, the audio noise of each step in the time step is obtained, and a noise vector is generated according to each of the audio noises, and the noise vector is denoised according to the first feature vector, the time step and the text command to obtain a target feature vector, and according to the target feature vector, a second audio is obtained, the second audio is composed of the audio sub-track corresponding to the first type in the first audio. In this process, the feature vector of the source audio and the text command used to indicate the type of audio sub-track are used to extract the audio sub-track of the source audio, so as to realize the customized separation and combined extraction of various musical instruments and human voices in the audio, and achieve the technical effect of reducing costs and increasing efficiency; and considering the situation where the source audio quality is low or damaged, denoising is performed during the audio sub-track extraction process, which can effectively improve the accuracy of audio separation. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:
[0037] Figure 1 The following schematically shows an application scenario of the audio track extraction method according to an embodiment of the present disclosure;
[0038] Figure 2 The following is a schematic diagram of a flow chart of an audio sub-track extraction method provided by an embodiment of the present disclosure;
[0039] Figure 3 Schematically shows Figure 2 Schematic diagram of the process of step S203;
[0040] Figure 4 The structure of the text encoding module in one embodiment of the present disclosure is schematically shown;
[0041] Figure 5 The structure diagram of the audio generation module in one embodiment of the present disclosure is schematically shown;
[0042] Figure 6The schematic diagram of the structure of the adapter module in one embodiment of the present disclosure is shown;
[0043] Figure 7a One of the flowcharts of another audio track division method provided by an embodiment of the present disclosure is schematically shown;
[0044] Figure 7b The second flowchart of another audio track division method provided by an embodiment of the present disclosure is schematically shown;
[0045] Figure 8 The structure diagram of the setting model in one embodiment of the present disclosure is schematically shown;
[0046] Figure 9a One of the schematic diagrams of the tag structure in an embodiment of the present disclosure is schematically shown;
[0047] Figure 9b The second schematic diagram schematically shows a tag structure in an embodiment of the present disclosure;
[0048] Fig.9c A schematic diagram schematically shows a forward extraction method in an embodiment of the present disclosure;
[0049] Figure 9d A schematic diagram schematically shows a reverse extraction method in an embodiment of the present disclosure;
[0050] Fig.10 A schematic diagram of a flow chart of an audio track division method provided by another embodiment of the present disclosure is shown schematically;
[0051] Fig.11 The structure diagram of a computer-readable storage medium provided by an embodiment of the present disclosure is schematically shown;
[0052] Fig.12 The structure diagram of an audio track splitting device provided by an embodiment of the present disclosure is schematically shown;
[0053] Fig.13 A schematic diagram of the structure of a computing device provided by an embodiment of the present disclosure is schematically shown;
[0054] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION
[0055] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.
[0056] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.
[0057] According to an embodiment of the present disclosure, an audio track extraction method, apparatus, medium and computing device are proposed.
[0058] In this article, it is important to understand that the terms used have the following meanings:
[0059] Diffusion model: A deep learning model architecture that learns the latent structure of a data set by modeling the diffusion of data points in the latent space, thereby achieving the goal of generating target data samples from noise. It is widely used in image generation and audio generation tasks.
[0060] Variational Autoencoder (VAE): It consists of two parts: encoder and decoder. The encoder compresses the input data into a latent vector, and the decoder generates new data from this latent vector.
[0061] Hidden vector: The VAE encoder compresses and reduces the dimension of a specific content into a vector, which facilitates calculation and reduces the amount of calculation. It can be restored to specific content such as images and audio through the VAE decoder.
[0062] Sound source separation: Analyze and extract the audio work, and output the separate tracks of different instruments (or other different audio components) in the audio work, such as guitar, electric guitar, violin, etc.
[0063] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0064] In addition, any number of elements in the drawings is for illustration and not limitation, and any naming is only for distinction and does not have any limiting meaning.
[0065] In addition, the data involved in the present disclosure may be data authorized by the user or fully authorized by all parties, and the collection, dissemination, and use of the data shall comply with the requirements of relevant laws and regulations. The implementation methods / examples of the present disclosure may be combined with each other.
[0066] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION
[0068] The inventors have found that in audio processing scenarios such as music production and post-processing, music information retrieval, and music copyright protection and management, it is necessary to separate the audio source to obtain different types of music tracks, and use the music tracks to perform audio processing in the form of related music production, retrieval, analysis and comparison. Therefore, the accuracy of audio tracks plays an important role in the above audio processing process.
[0069] In the related art, the sound source separation technology for extracting audio tracks is mainly implemented based on convolutional neural networks. For example, firstly, based on a human voice extraction model, human voice is extracted from the mixed audio to obtain an intermediate audio, which includes human voice audio frames and non-human voice audio frames, and then based on a human voice filtering model, the non-human voice audio frames of the intermediate audio are filtered out to obtain the final human voice audio; or, firstly, the mixed audio sampling sequence is subjected to discrete Fourier transform to obtain a frequency domain signal, the frequency domain signal is modulo-modulated to obtain an initial spectrogram, the frequency domain signal is phase-modulated to obtain an initial phase graph, and then the initial spectrogram is input into a convolutional neural network to obtain a human voice mask, and the human voice mask is dot-multiplied with the original spectrogram to obtain a human voice spectrogram; finally, the human voice spectrogram and the initial phase graph are used as the frequency domain signals of the human voice, and an inverse Fourier transform is performed to obtain a time domain human voice signal, i.e., the separation result; or, sound source separation is performed on the time domain audio, such as based on a multi-scale time domain audio operation method and using a time domain convolution (Temporal Convolutional The above-mentioned sound source separation scheme mainly relies on deep learning models. These models are often limited to separating specific music tracks, such as vocal tracks, guitar tracks, etc. during the definition and training process. This fixed training method leads to the limitation of model functions, that is, each model can only separate specific instruments or sound components. If different instruments or sound components need to be combined and customized, a model needs to be trained for each instrument or sound component separately, which undoubtedly increases the development and use costs, and each sub-track model is trained independently, which is not conducive to the improvement of the effect. In addition, the above-mentioned sound source separation scheme relies too much on special sub-track training data sets, which usually contain sub-track information of different instruments or sound components to guide the training process of the model. However, it is extremely difficult to obtain sub-track data sets, which not only requires professional recording equipment and environment, but also requires fine editing and processing of audio signals. Therefore, the available sub-track training data sets are relatively small, which is difficult to meet the needs of large-scale training, and the scarcity of such data sets leads to the lack of rich music prior knowledge in the training process of the model, thereby limiting its separation effect and generalization ability.
[0070] In addition to the above-mentioned sound source separation technology based on convolutional networks, in the direction of artificial intelligence (AI) music generation, with the rapid development of diffusion models in the field of images, AI music generation technology based on diffusion models has also been continuously proposed and upgraded. For example, the music generation large model technology based on text description can generate high-quality music works with high sampling rates. After the diffusion model (Riffusion) fine-tunes the AI text-to-image tool (Stable Diffusion) generated by the image, the spectrum information of the music can be obtained by inputting the text description; for another example, the audio generation technology based on the latent space diffusion model (AudioLDM) is implemented based on the latent space diffusion model. After the spectrum of the music audio is generated based on the text, the high-fidelity generative adversarial network (HiFi-GAN) is used to complete the conversion of the spectrum to audio; and the AI audio generation model (StableAudio), which is also based on the diffusion model and works in the latent space, realizes the generation of high-sampling-rate stereo music of any length by introducing time coding. The above-mentioned text-guided music generation technology can currently generate high-quality music audio, but the current generation link can only ensure that the generated music audio is consistent with the input text in macro-semantics, and cannot meet the purpose of extracting specific audio tracks from the source audio in the sound source separation task. Moreover, the separated instrument tracks should be highly consistent with the corresponding costs in the source audio, which is also something that the current large music generation model cannot do.
[0071] In view of this, the embodiment of the present disclosure provides a technical solution by obtaining the first audio and text command of the audio track to be extracted, and determining the first feature vector of the first audio, the text command is used to indicate the first type of the audio track extracted from the first audio, based on the set time step, obtaining the audio noise of each step in the time step, and generating a noise vector according to each of the audio noises, and denoising the noise vector according to the first feature vector, the time step and the text command to obtain the target feature vector, and according to the target feature vector, obtaining the second audio, the second audio is composed of the audio track corresponding to the first type in the first audio. In this process, the feature vector of the source audio and the text command used to indicate the type of audio track are used to extract the source audio, so as to realize the customized separation and combined extraction of various musical instruments and human voices in the audio, and achieve the technical effect of reducing costs and increasing efficiency; and considering the situation where the source audio quality is low or damaged, denoising is performed during the audio track extraction process, which can effectively improve the accuracy of audio separation.
[0072] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.
[0073] Application Scenario Overview
[0074] First reference Figure 1 The application scenarios of the solutions provided in the present disclosure are illustrated by examples. Figure 1 A possible application scenario diagram provided by the embodiment of the present disclosure is as follows: Figure 1 As shown, in this application scenario, the user opens an audio program on the terminal device 101, such as a music application. The user can initiate an audio sub-track extraction request to the server 102 through the terminal device 101 according to the demand. The request can carry the first audio of the audio sub-track request. After receiving the request, the server 102 extracts the audio sub-track of the first audio to obtain the second audio, and feeds the second audio back to the terminal device 101. The terminal device 101 processes the second audio in the music application, such as audio production or post-processing. Optionally, during the extraction process of the audio sub-track, the server 102 can obtain the denoised target feature vector according to the feature vector of the first audio and the text command (indicating the first type of the audio sub-track extracted from the first audio) to obtain the second audio extracted by the audio sub-track. Optionally, during the extraction process of the audio sub-track, the server 102 can also obtain the corresponding model (such as the setting model, the variational self-encoder, etc.) from the server 103, and use the corresponding model to extract the audio sub-track. These models are trained in the server 103. Exemplarily, the music prior knowledge of the music generation large model can be used to train the setting model.
[0075] Terminal devices may include, but are not limited to, computers, smart phones, tablet computers, e-book readers, Moving Picture experts group audio layer III (MP3) players, Moving Picture experts group audio layer IV (MP4) players, portable computers, car computers, wearable devices, desktop computers, set-top boxes, smart TVs, etc.
[0076] In this application scenario, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.
[0077] It should be noted that Figure 1 This is only a schematic diagram of an application scenario provided by the embodiment of the present disclosure. Figure 1 does not limit the equipment included in Figure 1 The positional relationship between the devices in the Figure 1 In the application scenario shown, a data storage device may also be included. The data storage device may be an external memory relative to the server 102 or an internal memory integrated in the server 102 .
[0078] Exemplary Methods
[0079] Combine the following Figure 1 For application scenarios, refer to Figure 1 To describe the audio repair method according to an exemplary embodiment of the present disclosure. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in this regard. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.
[0080] Please refer to Figure 2 , Figure 2 A flowchart of an audio track extraction method provided by an embodiment of the present disclosure is shown. The execution subject of the method may be a server 102. Figure 2 As shown, the method may include steps S201-204:
[0081] Step S201, obtaining a first audio of an audio track to be extracted and a text command, and determining a first feature vector of the first audio, wherein the text command is used to indicate a first type of an audio track to be extracted from the first audio.
[0082] Exemplarily, the first audio and text command of the audio sub-track to be extracted can be obtained from the user terminal, and the user can select the first audio of the audio sub-track to be extracted according to actual needs in the user terminal, wherein the first audio and text command can be obtained from the graphical user interface (GUI) option, such as the user can select predefined options in the graphical user interface. For example, in an audio processing software, the user can click a button or select a menu item to select the audio and audio type to be extracted from the audio sub-track, such as "human voice", "instrument", "background noise" of audio a, or, obtain it from the command running parameters, in the command line interface, the user can specify the audio type to be extracted by parameters, and can also obtain it by the way of configuration files, that is, the user can specify the audio type to be extracted by editing the configuration file, and the configuration file can be JSON, XML or other formats. In some examples, the execution subject of the audio sub-track extraction method can also be the user terminal, wherein the text command can also be obtained by issuing a natural language command from the user, for example, the user can enter a command by natural language: "extract the human voice part in the audio" or "extract background music from the audio", etc.
[0083] By way of further example, when a user needs to extract multiple types of audio tracks from a first audio, a text command can be used to indicate the extraction of multiple types of audio tracks from the first audio. For example, a first type of audio track to be extracted from the first audio carried in a text command includes multiple types required by the user (i.e., the first type may include multiple types of audio tracks that the user needs to extract), or, the number of the text commands may be multiple, each of which is used to indicate the extraction of one type of audio track (i.e., the first type corresponds to one type). In actual applications, users can make adaptive adjustments based on actual needs.
[0084] After obtaining the first audio, in order to facilitate the extraction and denoising of the audio sub-tracks, by determining the feature vector of the first audio, the feature vector can also refer to a multidimensional feature vector (i.e., a feature vector that has not been reduced in dimension), wherein the process of obtaining the first feature vector can be to extract the time domain features and frequency features in the first audio, including the extraction of time domain features, frequency domain features, time-frequency domain features, etc., wherein the time domain features of the audio signal are extracted, such as zero crossing rate, energy, short-time energy, etc., or the frequency domain features are extracted using Fourier transform (FFT), such as spectrum, spectrum centroid, spectrum bandwidth, etc.; or the time-frequency domain features are extracted using short-time Fourier transform (STFT) or Mel-frequency cepstrum coefficient (MFCC), and the extracted features are combined into a multidimensional vector as the feature vector of the first audio. Alternatively, other methods can be used to extract feature vectors and audio description information, for example, a deep learning model can be used to determine the feature vector of the first audio, etc.
[0085] Step S202: based on the set time step, obtain the audio noise of each step in the time step, and generate a noise vector according to each audio noise.
[0086] In this embodiment, a time step is the minimum time unit for audio processing. Each time step represents a discrete time point or time period, at which corresponding audio features, such as notes, chords, rhythms, etc., can be generated or predicted. In this case, those skilled in the art can adaptively set the step size of the time step in combination with actual applications, and this embodiment does not specifically limit this. By dividing the audio into multiple time steps, the audio data can be processed step by step to ensure that the features in each time period can be accurately extracted and processed.
[0087] In this embodiment, the audio noise at each time step is obtained and synthesized into a large noise vector to facilitate the denoising process in subsequent steps.
[0088] Step S203: De-noise the noise vector according to the first feature vector, the time step and the text command to obtain a target feature vector.
[0089] In this embodiment, the noise vector is denoised according to the first feature vector, time step and text command. In each time step, the feature information of the source audio and the text command for indicating the audio track are combined to guide the audio noise denoising and obtain a vector representation that retains the original features of the audio (for each audio track).
[0090] Exemplarily, a noise vector (Zt) can be constructed by selecting a time step (t) and input into a set model together with a text command (Ctext). The set model can be obtained based on diffusion model training. After t time steps of iteration, the latent vector Z0 of the target audio track can be obtained, that is, the target feature vector.
[0091] Alternatively, if Figure 3 As shown, the above step S203 denoises the noise vector according to the first feature vector, the time step and the text command, and may include the following steps:
[0092] Step S2031, determining a second eigenvector corresponding to the time step and a third eigenvector corresponding to the text command.
[0093] In this embodiment, the time step and the text command are used as features independent of the audio features (the first feature vector) and as guide vectors for denoising.
[0094] In this embodiment, the text command can be an encoded text vector. Exemplarily, the text prompt is encoded using a text encoding module to convert the text input into a text vector containing rich semantic information. The text encoding module can use contrastive language-audio pretraining (CLAP) and text-to-text transfer transformer (T5), where CLAP is specially trained for multimodal matching of audio and text. This embodiment can use a CLAP model such as Figure 4 As shown, in some embodiments, other text encoders may also be used.
[0095] Step S2032: denoise the noise vector a target number of times according to the first eigenvector, the second eigenvector, and the third eigenvector to obtain a target eigenvector, where the target number is the same as the number of the audio noise.
[0096] In this embodiment, the audio noise can be randomly introduced noise (such as noise introduced in the diffusion process of audio data using a diffusion model), or it can be the noise of the audio data itself. Generally speaking, there is a random noise at each time step. This embodiment denoises the noise vector in accordance with the amount of audio noise to achieve audio noise denoising for each audio sub-track type, so as to improve the denoising effect.
[0097] In some embodiments, denoising can be performed in a loop traversal manner to improve the denoising effect. Specifically, in the above steps, the noise vector is denoised for a target number of times according to the first eigenvector, the second eigenvector and the third eigenvector, and the following method can be used: obtain the first noise vector after the last denoising, and determine the intermediate vector according to the first eigenvector and the first noise eigenvector; input the intermediate vector, the second eigenvector and the third eigenvector into the adapter module; input the output parameters of the adapter module, the first noise vector, the second eigenvector and the third eigenvector into the audio generation module to obtain the second noise vector output by the audio generation module, and update the current denoising times; in response to the denoising times being less than the target times, update the second noise vector to the first noise vector, and return to execute the step of determining the intermediate vector according to the first eigenvector and the second eigenvector.
[0098] In this embodiment, for each denoising, an intermediate vector is determined by the denoised noise vector and the first eigenvector. The intermediate vector is the eigenvector of the noise that has undergone the previous denoising. The intermediate vector and the second eigenvector corresponding to the time step and the third eigenvector corresponding to the text command are input into the adapter module. The adapter module performs synchronous reasoning with the audio generation module and incorporates the reasoning results into the steps of the audio generation module. Through the customized adapter module, the detailed content information of the source audio can be extracted to guide the audio generation, so as to enable the audio generation model to generate high-quality audio that is consistent with the specified type of instrument content in the source audio.
[0099] In this embodiment, Zx and Ctext are input to the adapter module together. The adapter module generates a guidance vector of the same dimension as the audio generation module in each time step, guiding the audio generation module to extract and generate instrument tracks that conform to the text commands from the mixed audio. The audio generation module can be implemented based on a pre-trained music generation model, such as a text-generated audio model (StableAudio) or other models.
[0100] Furthermore, in the above steps, the intermediate vector can be determined based on the first eigenvector and the first noise eigenvector in the following manner: a convolution operation with an initial zero setting is performed on the first eigenvector to obtain a fourth eigenvector; and the fourth eigenvector and the first noise vector are added to obtain the intermediate vector.
[0101] In this example, for each intermediate vector input into the adapter, the feature vector can be preprocessed by the initial zero convolution operation, and some unnecessary features can be eliminated or the features can be standardized. The intermediate vector is obtained by convolving the fourth feature vector after the initial zero convolution operation with the first noise vector. The intermediate vector contains the mixed information of the original feature and the noise, which can be more suitable for the subsequent denoising process, and will not introduce other mixed information.
[0102] Through the above method, the feature vector of the audio and the noise vector can be superimposed and then input into the adapter for processing, and the feature information of the source audio can be retained, and the introduction of other error terms can be avoided.
[0103] In some embodiments, the adapter module may include a plurality of control units connected in sequence and a convolutional layer initially set to zero corresponding to each of the control units; the audio generation module includes a plurality of encoder units connected in sequence and a decoder unit corresponding to each of the encoder units, and each of the decoder units is connected in sequence;
[0104] Wherein, the output parameter of a control unit is input to the next control unit, and the output parameter of the control unit is input to the convolution layer corresponding to the control unit;
[0105] The output parameters of the convolutional layer are input to the decoder unit corresponding to the convolutional layer;
[0106] The output parameter of one encoder unit is input to the next encoder unit, and the output parameter of the one encoder unit is input to the decoder unit corresponding to the one encoder unit;
[0107] The output parameters of one decoder unit are input to the next decoder unit.
[0108] Exemplarily, the audio generation module can be implemented based on a large text music generation model (such as Diffusion, stableaudio, suno, etc.), which has the ability to generate music, input the desired music text description and noise, and generate high-quality complete music. Take stableaudio as an example to illustrate the audio generation module.
[0109] like Figure 5As shown, the audio generation module can be implemented using the Diffusion architecture, which consists of 4 levels of symmetrical downsampling encoder blocks (EncoderBlock) and upsampling decoder blocks (DecoderBlock), with jump connections between the downsampling encoder blocks and upsampling decoder blocks. Each block consists of 2 convolutional residual layers, followed by a series of self-attention and cross-attention layers. During inference, the text command prompt and time step information t are converted into vectors and input into the diffusion network (DiffusionUnet) together with the noise input Zt, where Zt is initially a random latent vector calculated based on t time steps, and each subsequent time step is output by DiffusionUnet Zt-1. After T time steps, the final audio latent vector Z0 is obtained and converted to audio representation (i.e., the second audio) through the VAE decoder.
[0110] For further example, in order to facilitate the audio generation module to generate a specified sub-track audio from the mixed audio combined with the text command, the present embodiment uses an adapter module to guide the generation process of the audio generation module. The adapter can identify the content in the text command and the mixed audio, and achieve fine control generation at the sampling point level. Figure 6 As shown, the adapter module may include a control module (ControlBlock) with the same structure as the encoder module in the diffusion network (DIffusionUnet) and a zero convolution (ZeroConv) with all initial weights set to 0. The latent vector Zx of the source audio is processed by ZeroConv and added to Zt, and input into the four cascaded ControlBlocks together with the prompt. The output of each layer of ControlBlock is processed by ZeroConv and added to the corresponding EncoderBlock of DiffusionUnet as the input of DecoderBlock, realizing the embedding of the content information of the source audio into the generation link of the audio generation module. Like DiffusionUnet, ControlNet will act on each of the T time steps.
[0111] Through the above technical solution, the above combined structure between the audio generation module and the adaptation module can achieve fine control and extraction of audio track features at the sampling point level.
[0112] Continue to refer to Figure 2 Step S204: Acquire a second audio according to the target feature vector, where the second audio is composed of audio tracks corresponding to the first type in the first audio.
[0113] Exemplarily, after obtaining the target feature vector, a deep learning method, such as a variational autoencoder (VAE) or a generative adversarial network (GAN), can be used to obtain the corresponding audio signal according to the feature vector. In some embodiments, the second audio corresponding to the target feature vector can also be obtained by other inversion methods of extracting the first feature vector from the source audio (first audio), such as short-time Fourier transform (STFT), Mel spectrum coefficient (MFCC), etc. to extract the first feature vector, and MFCC can be used to reversely convert it back to Mel spectrum, and then converted back to linear spectrum. Alternatively, an inverse Fourier transform (such as inverse STFT) can be used to convert the spectrum data back to a time domain signal. The present disclosure does not specifically limit the process of obtaining the second audio.
[0114] Through the above technical scheme, the source audio is subjected to audio track extraction processing by utilizing the feature vector of the source audio and the text command for indicating the audio track type, thereby realizing customized separation and combined extraction of various musical instruments and vocals in the audio, achieving the technical effect of reducing costs and increasing efficiency; and considering the situation where the source audio quality is low or damaged, denoising is performed in the audio track extraction process, which can effectively improve the accuracy of audio separation.
[0115] Figure 7a This is a flowchart of another audio track division method provided by an embodiment of the present disclosure. On the basis of the above embodiment, in order to improve the generalization ability of data and optimize the effect of audio track extraction, the codec encoding latent vector of the model is set as the feature vector, and the target feature vector is decoded to obtain the second audio of the corresponding audio track. Specifically, in addition to the above steps S201-S204, the first feature vector of the first audio is determined in the above step S201 of this embodiment, which may include the following steps S2011 and step S2012, and step S204 is further divided into step S2041:
[0116] Step S2011, obtain the first audio of the audio track to be extracted and the text command, and control the encoder in the set model to encode the first audio into a latent vector in the latent space.
[0117] In this embodiment, the setting model can be any neural network model with an encoder and a decoder. By controlling the encoder in the setting model, the first audio to be repaired is input into the encoder, and in the latent space, the encoder encodes the first audio into a latent vector.
[0118] It can be understood that the latent space is a low-dimensional representation space that can be used to capture the potential structure and characteristics of input data. This embodiment uses the latent space to encode the latent vector in the set model to achieve compression and abstraction of the original audio.
[0119] Step S2012: Determine the latent vector as the first feature vector.
[0120] In this embodiment, by taking the latent vector as the first feature vector of the audio, based on low-dimensional feature representation, subsequent audio denoising processing, audio track separation processing, etc. can be facilitated.
[0121] Step S2041: Based on the decoder in the set model, decode the target feature vector to obtain the second audio.
[0122] For example, the model is set to use a variational autoencoder as an example. Figure 8 As shown, it includes an encoder and a decoder. The encoder converts the audio signal x (i.e., the first audio) to the latent space representation z (i.e., the first feature vector). After the latent space completes the extraction and generation of the audio sub-track, the decoder converts the target feature vector z representation of the latent space back to the audio signal x' (the second audio), which can significantly improve the working efficiency of the model.
[0123] It is understandable that the above-mentioned encoder and decoder can be a deep learning model, and its input and output can be an audio signal in the time domain or a spectrum after Fourier transformation of the audio, depending on the implementation. In some embodiments, the encoder and decoder can be components of a large music generation model, and its parameters are trained in the large music generation model and used in conjunction with the music generation module.
[0124] Optionally, this embodiment also performs training of the setting model on the server side, specifically, as Figure 7b As shown, the training steps of setting the model also include the following steps S801 and S802.
[0125] Step S801: Acquire multiple training samples, where the training samples include a third audio and a label of each type of audio track in the third audio.
[0126] In one example, audio data may be collected from at least one audio data source to obtain training samples, such as multiple training samples may be obtained from an online audio library, a recording device (using high-quality and low-quality recording devices to record audio respectively), or an existing data set (using an existing annotated audio data set).
[0127] In another example, in order to make the audio track extraction more accurate, multiple training samples can also be obtained in the following manner: obtain multiple fourth audios and audio track extraction commands; based on the audio track extraction command, extract the second type of audio tracks specified by the audio track extraction command from at least one fourth audio; construct a fifth audio corresponding to the audio track extraction command based on the second type of audio tracks; determine the label that matches the second type of audio track in the fifth audio from each preset label as the label corresponding to the fifth audio, and obtain the training sample.
[0128] It should be noted that the first type and the second type in this embodiment may refer to the same type or different types. Similarly, the first audio, the second audio, etc. may be the same audio or different audio, and this embodiment of the disclosure does not specifically limit this.
[0129] In this example, each of the preset tags includes tags of multiple levels of the same type, and a tag of a previous level serves as a parent tag of a tag of a next level.
[0130] For example, each audio may have multiple tags, such as a first-level accompaniment tag, a second-level guitar tag, a string instrument tag, a third-level acoustic guitar tag, a bass tag, etc. Among them, the first-level accompaniment tag is the parent tag of the second-level guitar tag, and the second-level accompaniment tag is the parent tag of the second-level guitar tag. For further example, the tag structure can be as follows: Figure 9a , Figure 9b As shown, by defining the instrument label structure, it can be used as an available instrument selection in the text command. Among them, the first level is divided into vocals and accompaniment, among which the second level of vocals includes main voice, harmony, male voice, female voice and child voice. The second level of accompaniment includes percussion, bowed string instruments, plucked string instruments, keyboard, brass instruments, woodwind instruments and guitar. The third level of accompaniment is a more refined instrument division, such as percussion instruments can be further divided into timpani, snare drum, triangle, etc. These labels are not fixed and can continue to be expanded. At the same time, the labels are not necessarily mutually exclusive. For example, the main voice can be either male or female, and the acoustic guitar can belong to the guitar category in the second level or the string instrument category. Next, by collecting instrument audio data and assigning labels, each audio can have multiple labels, such as the first level accompaniment label, the second level guitar label, the string instrument label, the third level acoustic guitar label, the bass label, etc.
[0131] In order to obtain as much sample data as possible while reducing the standard difficulty, the present embodiment can obtain data (i.e., the fourth audio) in the following manner: the fourth audio can be obtained by obtaining the fourth audio from the network, synthesizing the fourth audio based on a timbre synthesis method, or generating the fourth audio based on an audio generation model, or generating the fourth audio based on a timbre synthesis method and an audio generation model.
[0132] Specifically, data with instrument labels can be obtained based on existing music libraries or public music data sets, and the instrument labels can be matched with the label system of this embodiment (for example, by establishing a mapping relationship); or, the audio of a specified instrument can be synthesized based on the Musical Instrument Digital Interface (MIDI) plus a timbre library synthesis method; or, based on the current large model of multiple music generation, pure music audio of a specified instrument can be generated, and so on.
[0133] In some embodiments, based on the audio track extraction command, the second type of audio tracks specified by the audio track extraction command are extracted from at least one fourth audio. The following method can be used: determine each of the second types based on the text information in the audio track extraction command; and extract the audio tracks corresponding to each of the second types in at least one of the fourth audios.
[0134] In this example, the audio sub-track extraction command may include descriptive text information, which can be used to identify and classify the types of audio sub-tracks. For example, the command may include "human voice", "background music" or "ambient sound", etc. These descriptive text information can help the system determine the second type of each audio sub-track. Among them, the fourth audio can be a composite audio file containing multiple audio sub-tracks. The system can find and extract the corresponding audio sub-track in this composite audio file based on the second type determined in the previous step. For example, if the second type is "human voice", the audio sub-track containing the human voice is found and extracted in the fourth audio; if the second type is "background music", the audio sub-track containing the background music will be extracted.
[0135] In this way, the text information in the command can be extracted according to the audio track, and the required audio track can be accurately identified and extracted, thereby realizing the classification and processing of the audio content.
[0136] Furthermore, the type of the audio track can be determined by forward extraction and reverse extraction to improve the efficiency of obtaining training samples. Specifically, the above-mentioned determination of each second type based on the text information in the audio track extraction command can be carried out in the following manner: determining the semantic information of the audio track extraction command; in response to the semantic information indicating that the audio track extraction command is a forward extraction command, determining the type of the audio track contained in the text information in the audio track extraction command as the second type; and / or, in response to the semantic information indicating that the audio track extraction command is a reverse extraction command, determining the type of the audio track not contained in the text information in the audio track extraction command as the second type.
[0137] For example, after the audio data set is constructed, the text commands and corresponding audio data can be dynamically and randomly constructed during the training process based on a preset text template (which can be adaptively determined by those skilled in the art in combination with actual applications). Fig.9c and Figure 9d As shown, the following methods may be included:
[0138] Forward instrument extraction: For example, if you want to extract vocals and guitar from audio, the data construction rule is to extract an audio V with only vocals as the first-level label and an audio J with only guitar as the second-level label from the data set, and then randomly extract i (>=0) audios, requiring that the first-level label does not contain vocals, the second-level label does not contain guitar, and the same third-level label is not repeated. Add up all the audios as the mixed audio x, add up audio V and audio J as the target t, and the prompt text as prompt to form a set of training data.
[0139] Reverse instrument extraction: For example, "remove vocals and guitars from audio" and "extract other instruments except vocals and guitars from audio", the data construction rules are the same as above. Add up all the audios as mixed audio x, add up the audios except audio V and audio J as target t, and use the prompt text as prompt to form a set of training data.
[0140] In some examples, forward instrument extraction and reverse instrument extraction may also be performed simultaneously to form corresponding training samples.
[0141] Continue to refer to Figure 8 , step S802, training the preset model according to each of the training samples to obtain a set model.
[0142] Exemplarily, the loss function of the preset model in this embodiment can be updated based on the reconstruction loss function, the adversarial loss function, the adversarial feature loss function, the divergence loss function, and the third audio.
[0143] It is understood that in machine learning and deep learning, the loss function is a function used to measure the difference between the model's predicted results and the true results.
[0144] In this example, for the preset model, the loss function can be designed and updated based on the following types: reconstruction loss function, which is used to measure the accuracy of the model when reconstructing the input data, and can include mean square error (MSE) and mean absolute error (MAE). In the audio track extraction task of this embodiment, the reconstruction loss function can be used to evaluate the difference between the audio generated by the model and the original audio; the adversarial loss function can be used in the generative adversarial network (GAN) to measure the game effect between the generator and the discriminator, which can use cross entropy loss. In the audio track extraction task, the generator tries to generate realistic audio clips, while the discriminator tries to distinguish between the generated audio and the real audio. The adversarial loss function helps the generator generate more realistic audio; the adversarial feature loss function is used to measure the difference between the generated features and the real features, which can be used to improve the feature representation of the generative model. For example, methods such as feature matching loss can be used. In the audio sub-track extraction task, this loss function can help the generator generate audio with specific features, such as a certain timbre or frequency feature; the divergence loss function is used to measure the difference between two probability distributions, wherein the divergence metric can include Kullback-Leibler divergence (KL divergence) and Jensen-Shannon divergence (JS divergence), etc. In the audio sub-track extraction of the present embodiment, the divergence loss function can be used to measure the difference between the generated audio distribution and the real audio distribution, thereby improving the performance of the generation model. In addition, the loss function can also be based on the update of the third audio, in other words, the loss function not only considers the current input and output, but also considers additional audio data. For example, in the audio sub-track fierce conversion task, the third audio can be a reference audio sub-track of the target type, and the loss function can measure the difference between the generated audio and the reference audio sub-track. In some embodiments, any one of the above or in combination with the above different types of loss functions can be used to design a more complex and effective preset model loss function, thereby improving the performance of the model in various audio sub-track extraction tasks.
[0145] Compared with the sound source separation method in the related art, which is limited by the limited track data set, these data sets often cannot fully cover the complexity and diversity of music. The disclosed embodiment combines the music generation large model with the sound source separation technology to form a new solution. By accumulating a large amount of music knowledge priors during the training process of the large model, including the recognition of the sounds of different instruments, the understanding of the music structure, the mastery of the sound mixing method, etc., when performing sound source separation, the large model can integrate these rich music knowledge priors into the model as prior knowledge to guide the model to more accurately separate the new audio data. This enables the model to accurately identify and separate different sound sources without a large amount of track data. Through this combination, an end-to-end sound source separation system can be constructed, which can not only automatically identify different sound sources in the audio, but also perform customized separation and combination according to the user's text input, which will bring more efficient and intelligent solutions to the fields of audio processing and music production.
[0146] In an exemplary embodiment, text commands are input from mixed music audio to extract the sub-track audio of any target. It can be based on a pre-trained music generation model (i.e., audio generation model) to make full use of the music prior data therein to achieve the recognition and high-quality extraction of different instrument tracks, and use preset text prompt templates and data composition methods to train setting models, adapter modules and other models, so that they can introduce text prompts into the generation process of the audio generation model, and guide the audio generation model to generate specified sub-track audio. Specifically, the network architecture of the technical solution provided in this embodiment may include: an audio latent space encoding and decoding module, an audio generation module, a text encoding module and an adapter module, and its exemplary processing flow is as follows: Fig.10 As shown:
[0147] First, input the text command prompt of the extraction track you want to extract, and you can use the pre-trained text encoder to encode it into a vector Ctext. Input the mixed audio x, and the audio latent space encoder (set model) encodes it into a latent space vector Zx (i.e., the first eigenvector). Select a time step t, construct a noise vector Zt, and input it into the audio generation module together with Ctext. After t time steps of iterative generation, the latent vector Z0 of the target track audio (i.e., the target eigenvector) is obtained. At the same time, Zx and Ctext are input into the adapter module together. The adapter module generates a guide vector of the same dimension as the generation module in each time step, guiding the audio generation module to extract and generate instrument tracks that meet the text command from the mixed audio. In the example, the text encoder, audio latent space codec, and audio generation module are implemented based on a music generation pre-training model, and their weights can be directly taken from the pre-training model and fixed to reuse the music knowledge prior in the pre-training model, where the setting model and adapter module can be trained based on the constructed sound source separation data set.
[0148] In summary, the audio track extraction scheme based on text commands in the disclosed embodiment can meet the input text commands and perform customized separation and combination extraction of various musical instruments and vocals in the audio; and by utilizing the sound source separation scheme of music prior knowledge in the pre-trained music generation large model, it can complete the task of high-quality specified instrument music extraction without a fine music track data set; and by utilizing a customized adapter module, it can extract detailed content information of the source audio, guide audio generation, and ensure that the audio generation model generates high-quality audio consistent with the specified instrument content in the source audio; in addition, through structured multi-level instrument labels, audio from multiple channels can be used to optimize the sound source separation effect, and forward and reverse sound source separation text instructions are proposed, and corresponding training data combination methods are given, thereby realizing training optimization of the entire audio track extraction process.
[0149] Exemplary Media
[0150] After introducing the method of the exemplary embodiment of the present disclosure, next, refer to Fig.11 A computer-readable storage medium according to an exemplary embodiment of the present disclosure is described.
[0151] refer to Fig.11 As shown, the storage medium 30 stores a program product for implementing the above method according to an embodiment of the present disclosure, which can be a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.
[0152] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0153] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable signal medium may also be any readable medium other than a readable storage medium.
[0154] Program code for performing the disclosed operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN).
[0155] Exemplary Devices
[0156] After introducing the medium of the exemplary embodiment of the present disclosure, next, reference is made to Fig.12 An audio repair device according to an exemplary embodiment of the present disclosure is described, which is used to implement a method in any of the above method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.
[0157] Fig.12 A schematic diagram of the structure of an audio track extraction device provided by an embodiment of the present disclosure is shown. Fig.12As shown, the device includes a first acquisition module 121, a second acquisition module 122, a denoising module 123 and a third acquisition module 124; wherein the first acquisition module 121 is used to acquire a first audio and a text command of an audio track to be extracted, and determine a first feature vector of the first audio, and the text command is used to indicate a first type of an audio track extracted from the first audio; the second acquisition module 122 is used to acquire the audio noise of each step in the time step based on a set time step, and generate a noise vector according to each audio noise; the denoising module 123 is used to denoise the noise vector according to the first feature vector, the time step and the text command to obtain a target feature vector; the third acquisition module 124 is used to acquire a second audio according to the target feature vector, and the second audio is composed of audio tracks corresponding to the first type in the first audio.
[0158] In one embodiment of the present disclosure, the first acquisition module 121 includes: a control unit, used to control the encoder in the set model to encode the first audio into a latent vector in the latent space; a determination unit, used to determine the latent vector as the first feature vector; the third acquisition module includes: a decoding unit, used to decode the target feature vector based on the decoder in the set model to obtain the second audio.
[0159] In another embodiment of the present disclosure, the training module of the set model includes: a sample acquisition unit, used to acquire multiple training samples, the training samples including a third audio and a label of each type of audio track in the third audio; a training unit, used to train a preset model according to each of the training samples to obtain the set model.
[0160] In another embodiment of the present disclosure, the loss function of the preset model is updated based on the reconstruction loss function, the adversarial loss function, the adversarial feature loss function, the divergence loss function, and the third audio.
[0161] In another embodiment of the present disclosure, the sample acquisition unit is specifically used to: acquire multiple fourth audios and audio track extraction commands; based on the audio track extraction command, extract each second type of audio track specified to be extracted by the audio track extraction command from at least one fourth audio; construct a fifth audio corresponding to the audio track extraction command according to each second type of audio track; determine, from each preset label, a label matched by the second type of audio track in the fifth audio as the label corresponding to the fifth audio, and obtain the training sample.
[0162] In yet another embodiment of the present disclosure, based on the audio track extraction command, extracting each second type of audio track specified by the audio track extraction command from at least one fourth audio, specifically: determining each second type according to text information in the audio track extraction command; and extracting each audio track corresponding to the second type in at least one of the fourth audios.
[0163] In still another embodiment of the present disclosure, the determining of each second type according to the text information in the audio track extraction command is specifically as follows: determining the semantic information of the audio track extraction command; in response to the semantic information indicating that the audio track extraction command is a forward extraction command, determining the type of the audio track contained in the text information in the audio track extraction command as the second type; in response to the semantic information indicating that the audio track extraction command is a reverse extraction command, determining the type of the audio track not contained in the text information in the audio track extraction command as the second type.
[0164] In yet another embodiment of the present disclosure, the obtaining of multiple fourth audios specifically includes: obtaining the fourth audio from a network, synthesizing the fourth audio based on a timbre synthesis method, and / or generating the fourth audio based on an audio generation model.
[0165] In yet another embodiment of the present disclosure, each of the preset tags includes tags of multiple levels of the same type, and a tag of an upper level serves as a parent tag of a tag of a lower level.
[0166] In another embodiment of the present disclosure, the denoising module 123 includes: a vector determination unit, used to determine the second eigenvector corresponding to the time step and the third eigenvector corresponding to the text command; a multiple denoising unit, used to denoise the noise vector a target number of times according to the first eigenvector, the second eigenvector and the third eigenvector to obtain a target eigenvector, and the target number is the same as the amount of audio noise.
[0167] In another embodiment of the present disclosure, according to the multiple denoising units, it is specifically used to: obtain the first noise vector after the last denoising, and determine the intermediate vector according to the first eigenvector and the first noise eigenvector; input the intermediate vector, the second eigenvector and the third eigenvector into the adapter module; input the output parameters of the adapter module, the first noise vector, the second eigenvector and the third eigenvector into the audio generation module to obtain the second noise vector output by the audio generation module, and update the current denoising times; in response to the denoising times being less than the target times, update the second noise vector to the first noise vector, and return to execute the step of determining the intermediate vector according to the first eigenvector and the second eigenvector.
[0168] In still another embodiment of the present disclosure, the determining of the intermediate vector based on the first eigenvector and the first noise eigenvector is specifically as follows: performing a convolution operation with an initial zero setting on the first eigenvector to obtain a fourth eigenvector; and adding the fourth eigenvector and the first noise vector to obtain the intermediate vector.
[0169] In another embodiment of the present disclosure, the adapter module includes a plurality of control units connected in sequence and a convolutional layer initially set to zero corresponding to each of the control units; the audio generation module includes a plurality of encoder units connected in sequence and a decoder unit corresponding to each of the encoder units, and the decoder units are connected in sequence; wherein the output parameters of a control unit are input to the next control unit, and the output parameters of the control unit are input to the convolutional layer corresponding to the control unit; the output parameters of the convolutional layer are input to the decoder unit corresponding to the convolutional layer; the output parameters of an encoder unit are input to the next encoder unit, and the output parameters of the encoder unit are input to the decoder unit corresponding to the encoder unit; the output parameters of a decoder unit are input to the next decoder unit.
[0170] Exemplary Computing Devices
[0171] After introducing the method, medium and apparatus of the exemplary embodiments of the present disclosure, next, reference is made to Fig.13 A computing device according to an exemplary embodiment of the present disclosure is described.
[0172] Fig.13 The computing device 40 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present disclosure. Fig.13As shown, the computing device 40 is in the form of a general computing device. The components of the computing device 40 may include, but are not limited to: at least one processing unit 401, at least one storage unit 402, and a bus 403 connecting different system components (including the processing unit 401 and the storage unit 402). Among them, at least one storage unit 402 stores computer-executable instructions; at least one processing unit 401 includes a processor, and the processor executes the computer-executable instructions to implement the method described above. The bus 403 includes a data bus, a control bus, and an address bus. The storage unit 402 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 4021 and / or a cache memory 4022, and may further include a readable medium in the form of a non-volatile memory, such as a read-only memory (ROM) 4023.
[0173] The storage unit 402 may also include a program / utility 4025 having a set (at least one) of program modules 4024, such program modules 4024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include the implementation of a network environment. The computing device 40 may also communicate with one or more external devices 404 (e.g., a keyboard, a pointing device, etc.). Such communication may be performed via an input / output (I / O) interface 405. Furthermore, the computing device 40 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 406. Fig.13 As shown, network adapter 406 communicates with other modules of computing device 40 via bus 403. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with computing device 40, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0174] It should be noted that although several units / modules or sub-units / modules of the audio repair device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided to be embodied by multiple units / modules.
[0175] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.
[0176] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.
Claims
1. A method for extracting audio tracks, characterized in that: include: Acquire a first audio of an audio track to be extracted and a text command, and determine a first feature vector of the first audio, wherein the text command is used to indicate a first type of an audio track to be extracted from the first audio; Based on a set time step, obtaining the audio noise of each step in the time step, and generating a noise vector according to each of the audio noises; De-noising the noise vector according to the first feature vector, the time step and the text command to obtain a target feature vector; According to the target feature vector, a second audio is obtained, where the second audio is composed of audio tracks corresponding to the first type in the first audio.
2. The audio track extraction method according to claim 1, characterized in that: The determining a first feature vector of the first audio comprises: Controlling an encoder in the setting model to encode the first audio into a latent vector in a latent space; Determine the latent vector as the first feature vector; The step of acquiring the second audio according to the target feature vector includes: Based on the decoder in the set model, the target feature vector is decoded to obtain the second audio.
3. The audio track extraction method according to claim 2, characterized in that: The training steps of the setting model are as follows: Acquire multiple training samples, where the training samples include a third audio and a label of each type of audio track in the third audio; The preset model is trained according to each of the training samples to obtain the set model.
4. The audio track extraction method according to claim 3, characterized in that: The loss function of the preset model is updated based on the reconstruction loss function, the adversarial loss function, the adversarial feature loss function, the divergence loss function, and the third audio.
5. The audio track extraction method according to claim 3, characterized in that: The obtaining of multiple training samples comprises: Get multiple fourth audio and audio track extraction commands; Based on the audio track extraction command, extracting, from at least one fourth audio, respective second-type audio tracks specified to be extracted by the audio track extraction command; Constructing a fifth audio corresponding to the audio track extraction command according to each of the second-type audio tracks; From various preset labels, determine the label that matches the second type of the audio track in the fifth audio as the label corresponding to the fifth audio, and obtain the training sample.
6. The audio track extraction method according to claim 5, characterized in that: The extracting, from at least one fourth audio, the respective second-type audio tracks specified to be extracted by the audio track extraction command based on the audio track extraction command, comprises: Determining each of the second types according to the text information in the audio track extraction command; In at least one of the fourth audios, extract each audio track corresponding to the second type.
7. The audio track extraction method according to claim 6, characterized in that: The step of determining each of the second types according to the text information in the audio track extraction command includes: Determining semantic information of the audio track extraction command; In response to the semantic information indicating that the audio track extraction command is a forward extraction command, determining the type of the audio track contained in the text information in the audio track extraction command to be the second type; and / or In response to the semantic information indicating that the audio track extraction command is a reverse extraction command, the type of the audio track not included in the text information in the audio track extraction command is determined to be the second type.
8. An audio track extraction device, characterized in that: include: A first acquisition module, used to acquire a first audio of an audio track to be extracted and a text command, and determine a first feature vector of the first audio, wherein the text command is used to indicate a first type of an audio track to be extracted from the first audio; A second acquisition module is used to acquire the audio noise of each time step based on the set time step, and generate a noise vector according to each audio noise; A denoising module, configured to denoise the noise vector according to the first feature vector, the time step and the text command to obtain a target feature vector; The third acquisition module is used to acquire a second audio according to the target feature vector, where the second audio is composed of audio tracks corresponding to the first type in the first audio.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 7 is implemented.
10. A computing device, characterized in that include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the computing device to perform the method as claimed in any one of claims 1 to 7.
Citation Information
Patent Citations
System and method for transmitting voice input from a remote location over a wireless data channel
CA2378535A1
Audio track switching method, system thereof and audio and video file playing device
CN102347042A
Audio separation method, device and equipment and computer readable storage medium
CN110827850A
Audio separation method and device, electronic equipment and computer readable storage medium
CN111724807A
Audio separation method and device, equipment, storage medium and program product
CN114365219A
Cited By
Method for repairing sound quality, electronic equipment, storage medium and program product
CN120510857A
Data processing method, data processing device, electronic equipment and storage medium
CN121096313A
Audio editing method, apparatus, device, storage medium and computer program product
CN122821984A
