Audio separation method, device, equipment, storage medium and program product

Abstract

The present application discloses an audio separation method, apparatus, device, storage medium and program product, which belongs to the field of audio processing technology. The method comprises: obtaining the audio to be separated, the audio to be separated includes at least two tracks; obtaining the time domain features and texture features of the audio to be separated, the time domain features and texture features are used to characterize the harmonic correlation and harmonic continuity of the audio to be separated respectively; according to the time domain features and texture features, obtaining the spectral features corresponding to n track sets respectively, the spectral features are used to characterize the frequency and amplitude information of the track sets, each track set includes a combination of several tracks in the audio to be separated; according to the spectral features corresponding to the n track sets respectively, n corresponding audio files are generated. The present application provides an audio separation method with good separation effect, which reduces the computational load of the equipment during the audio separation process and improves the speed of audio separation by generating the spectral features corresponding to the track sets respectively through the time domain features and texture features.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio separation method, apparatus, device, storage medium and program product. Background Art

[0002] Music is an audio file that is a mixture of human voice and the sounds of various instruments. Separating audio files to obtain multiple independent audio tracks in the audio file has important applications in music mixing, accompaniment extraction, etc.

[0003] In the related art, an audio separation method based on a convolutional neural network is used to separate the audio to be separated. When using this method to separate the human voice and the accompaniment sound, the audio to be separated is first input into the audio separation model, and the audio to be separated is convolved by the audio separation model to obtain the human voice features and the accompaniment features respectively, and the separated human voice track and accompaniment track are generated based on the separated human voice features and accompaniment features.

[0004] However, when the above audio separation method separates the audio tracks, the calculation amount of the separation process is large and the separation speed is slow. Summary of the Invention

[0005] The present invention provides an audio separation method, apparatus, device, storage medium, and program product. In the process of separating the audio to be separated to obtain multiple audio track sets, the computational complexity is small and the separation speed is fast. The technical solution is as follows:

[0006] According to one aspect of an embodiment of the present application, a method for audio separation is provided, the method comprising:

[0007] Acquire audio to be separated, where the audio to be separated includes at least two audio tracks;

[0008] Acquire a time domain feature and a texture feature of the audio to be separated, wherein the time domain feature is used to characterize the harmonic correlation of the audio to be separated, and the texture feature is used to characterize the harmonic continuity of the audio to be separated;

[0009] Obtaining, based on the time domain features and the texture features, spectral features corresponding to n audio track sets, respectively, wherein the spectral features are used to characterize frequency and amplitude information of the audio track sets, each audio track set including one audio track or a combination of multiple audio tracks of the audio to be separated, where n is a positive integer;

[0010] Audio files corresponding to the n audio track sets are generated according to the spectral features corresponding to the n audio track sets.

[0011] According to one aspect of an embodiment of the present application, a method for training an audio separation model is provided, the method comprising:

[0012] Obtaining training data for the audio separation model, the training data including an audio sample to be separated and n labeled audio tracks corresponding to the audio sample to be separated, the audio sample to be separated including at least two audio tracks, where n is a positive integer;

[0013] Acquiring, through the audio separation model, time domain features and texture features of the audio sample to be separated, wherein the time domain features are used to characterize the harmonic correlation of the audio sample to be separated, and the texture features are used to characterize the harmonic continuity of the audio sample to be separated;

[0014] Obtaining, based on the time domain features and the texture features, spectral features corresponding to n audio track sets, respectively, wherein the spectral features are used to characterize frequency and amplitude information of the audio track sets, each audio track set including one audio track or a combination of multiple audio tracks of the audio sample to be separated;

[0015] The training loss of the audio separation model is calculated according to the spectral features corresponding to the n audio track sets and the spectral features corresponding to the n labeled audio tracks, and the audio separation model is trained based on the training loss.

[0016] According to one aspect of an embodiment of the present application, there is provided an audio separation device, the device comprising:

[0017] An audio acquisition module, configured to acquire audio to be separated, wherein the audio to be separated includes at least two audio tracks;

[0018] a feature extraction module, configured to obtain time domain features and texture features of the audio to be separated, wherein the time domain features are used to characterize the harmonic correlation of the audio to be separated, and the texture features are used to characterize the harmonic continuity of the audio to be separated;

[0019] a spectrum generation module, configured to obtain spectrum features corresponding to n audio track sets based on the time domain features and the texture features, wherein the spectrum features are used to characterize frequency and amplitude information of the audio track sets, each audio track set including one audio track or a combination of multiple audio tracks of the audio to be separated, where n is a positive integer;

[0020] The audio track generation module is used to generate audio files corresponding to the n audio track sets according to the frequency spectrum features corresponding to the n audio track sets.

[0021] According to one aspect of an embodiment of the present application, a training device for an audio separation model is provided, the device comprising:

[0022] A data acquisition module, configured to acquire training data for the audio separation model, wherein the training data includes an audio sample to be separated and n labeled audio tracks corresponding to the audio sample to be separated, wherein the audio sample to be separated includes at least two audio tracks, and n is a positive integer;

[0023] a feature extraction module, configured to obtain time domain features and texture features of the audio sample to be separated through the audio separation model, wherein the time domain features are used to characterize the harmonic correlation of the audio sample to be separated, and the texture features are used to characterize the harmonic continuity of the audio sample to be separated;

[0024] a spectrum generation module, configured to obtain spectrum features corresponding to n audio track sets respectively based on the time domain features and the texture features, wherein the spectrum features are used to characterize frequency and amplitude information of the audio track sets, each audio track set including one audio track or a combination of multiple audio tracks of the audio sample to be separated;

[0025] A model training module is used to calculate the training loss of the audio separation model based on the spectral features corresponding to the n audio track sets and the spectral features corresponding to the n labeled audio tracks, and train the audio separation model based on the training loss.

[0026] According to one aspect of an embodiment of the present application, a computer device is provided, comprising: a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the audio separation method or the audio separation model training method.

[0027] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned audio separation method or audio separation model training method.

[0028] According to one aspect of an embodiment of the present application, a computer program product or a computer program is provided, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor reads the computer instructions from the computer-readable storage medium to implement the audio separation method or the audio separation model training method as described above.

[0029] The technical solutions provided in the embodiments of the present application can bring the following beneficial effects:

[0030] By obtaining the time domain features and texture features of the audio to be separated, and then performing audio separation based on these two features, since the time domain features and texture features only contain features related to harmonics and do not contain features related to factors such as phase in the audio to be separated, the computational cost of obtaining the time domain features and frequency domain features of the audio to be separated is small during the audio separation process. The time domain features and frequency domain features of the audio to be separated obtained by this method have smaller dimensions than the audio features obtained by directly convolving the audio to be separated. Therefore, the computational cost of audio separation by this method is small and the audio separation speed is fast. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0032] Figure 1 This is a schematic diagram of an implementation environment for a solution provided by an embodiment of the present application;

[0033] Figure 2 This is a flowchart of an audio separation method provided by an embodiment of the present application;

[0034] Figure 3 This is a schematic diagram of an audio separation process provided by an embodiment of the present application;

[0035] Figure 4 is a schematic diagram of an audio separation process provided by another embodiment of the present application;

[0036] Figure 5 is a flowchart of an audio separation method provided by another embodiment of the present application;

[0037] Figure 6 Schematic diagram of a network structure of an audio separation model provided in an embodiment of the present application;

[0038] Figure 7 Schematic diagram of another network structure of the audio separation model provided in an embodiment of the present application;

[0039] Figure 8 is a schematic diagram of an audio separation method provided by another embodiment of the present application;

[0040] Figure 9 This is a flowchart of a method for training an audio separation model provided by one embodiment of the present application;

[0041] Figure 10 is a schematic diagram of a training method for an audio separation model provided by an embodiment of the present application;

[0042] Figure 11 A block diagram of an audio separation device provided in one embodiment of the application;

[0043] Figure 12 A block diagram of a training device for an audio separation model provided in one embodiment of the application;

[0044] Figure 13 A schematic diagram of a computer device provided in one embodiment of the application. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0046] Please refer to Figure 1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment may include a model training device 10 and an audio processing device 20.

[0047] The model training device 10 is an electronic device for training an audio separation model, and may be an electronic device such as a PC (Personal Computer), a server, etc. The audio separation model trained by the model training device 10 may be deployed in the audio processing device 20 for use.

[0048] The audio processing device 20 is an electronic device for processing the audio to be separated. The audio processing device 20 can be an electronic device such as a mobile phone, a tablet computer, an intelligent robot, or a server. The audio processing device 20 can perform audio separation processing on the audio to be separated using an audio separation model, generating n audio track sets, and then obtaining audio files corresponding to each of the n audio track sets, where n is a positive integer. Optionally, the audio processing device 20 also has functions such as music playback and audio synthesis, which are not limited in this application.

[0049] The technical solution provided in the embodiments of the present application can be applied to any practical application scenario where there is a need to separate and process audio files.

[0050] The audio processing system may include a terminal device and a server. The terminal device has functions such as audio data storage, audio playback and data storage, and the server can provide background support for the audio processing function for the terminal device.

[0051] In one example, an audio separation system is installed on a terminal device, and the audio separation process is performed on the terminal device. After obtaining the audio to be separated, the terminal device performs feature extraction on the audio to obtain time domain features and texture features of the audio to be separated. Based on the time domain features and texture features of the audio to be separated, the spectral features corresponding to each of the n audio track sets are obtained. The terminal device then obtains the audio files of the n audio track sets based on the n frequency-amplitude features, completing the audio separation process. Optionally, if k of the audio files corresponding to the n audio track sets meet the needs of the mobile phone operator, where k is a positive integer less than or equal to n, the mobile phone operator will select only the k audio files that meet the needs for use.

[0052] In another example, an audio separation system is installed on a server, and the audio separation process is performed on the server. After the terminal device obtains the audio to be separated, it sends it to the server. The server receives the audio to be separated from the terminal device, extracts the time domain features and texture features of the audio to be separated, and then uses these time domain features and texture features to obtain the spectral features of n audio track sets. It then generates audio files corresponding to the n audio track sets. The server then sends the n audio files to the terminal device, completing the audio separation process.

[0053] Of course, the exemplary application scenarios introduced above are only some typical application scenarios introduced to facilitate understanding of the technical solution of this application. The technical solution of this application can also be applied to other actual application scenarios where there is a need to separate audio files, and the embodiments of this application are not limited to this.

[0054] Please refer to Figure 2 , which shows a flowchart of an audio separation method provided by an embodiment of the present application. The execution subject of each step of the method can be Figure 1 In the audio processing device 20 in the implementation environment of the illustrated solution, the method may include at least one of the following steps (210-240):

[0055] Step 210: Acquire audio to be separated, where the audio to be separated includes at least two audio tracks.

[0056] The audio to be separated refers to the audio file used for audio separation. An audio file refers to information obtained by sampling loudness in the time and frequency domains. An audio track records the relationship between a class of audio signals with the same attributes and time. The attributes of an audio track include timbre, timbre library, and input and output channels. Audio tracks include single tracks and multiple tracks. A single track is also called a mono signal track. For example, the audio of a recorded instrument belongs to a single track, and the audio of a person singing a cappella also belongs to a single track. Multiple tracks include multiple tracks obtained by superimposing multiple identical tracks, or multiple tracks obtained by superimposing multiple different tracks. The audio to be separated contains at least two tracks. For example, the audio to be separated is audio related to an instrumental ensemble, and the audio to be separated contains tracks corresponding to piano, violin, cello, flute, clarinet, tuba, and timpani. For another example, the audio to be separated is a song-type audio, which includes a vocal track and an accompaniment track, and the accompaniment track includes tracks corresponding to guitar, bass, electronic synthesizer, and drums, respectively. The type of audio to be separated and the types of tracks included in the audio to be separated are determined according to actual needs and are not limited here. The audio processing device obtains the audio to be separated, which is a mixed audio composed of multiple tracks, and the audio processing device is capable of performing audio separation on the audio to be separated.

[0057] Step 220: Acquire the time domain features and texture features of the audio to be separated. The time domain features are used to characterize the harmonic correlation of the audio to be separated, and the texture features are used to characterize the harmonic continuity of the audio to be separated.

[0058] The audio processing device analyzes the audio to be separated and obtains time domain features and texture features. The time domain features contain multiple different time domain feature information, and different audio tracks have different time domain feature information. The texture features contain multiple different texture feature information, and different audio tracks have different texture feature information. Sound appears as up and down vibrating trajectories on the time axis, and these trajectories are called harmonics. Different musical instruments and human voices have different timbre, frequency and other attributes, so different musical instruments and human voices have different harmonics. Texture features are used to represent the continuity of harmonics, that is, the changing patterns and characteristics of harmonics along the time axis. Time domain features are used to represent the correlation of harmonics, that is, the changing patterns and characteristics of harmonics up and down vibrations, as well as the changing patterns and characteristics along the time axis, in the time domain features.

[0059] The audio processing device obtains time domain features and texture features based on the frequency and amplitude features of the audio to be separated. The extracted time domain features and texture features are used to obtain the spectrum features of the audio track set.

[0060] The time domain features and texture features of the audio to be separated can be used to grasp the harmonic characteristics of the audio to be separated and the feature information of different audio track sets, which is conducive to the subsequent network generating the spectral features of n audio track sets through the time domain features and texture features of the audio to be separated.

[0061] Step 230: Obtain spectral features corresponding to n audio track sets based on the time domain features and texture features. The spectral features are used to characterize the frequency and amplitude information of the audio track sets. Each audio track set includes one audio track or a combination of multiple audio tracks in the audio to be separated. n is a positive integer.

[0062] A track set refers to the tracks obtained after an audio processing device separates the audio to be separated. In some embodiments, the track set is a track corresponding to a single instrument or vocals. In other embodiments, the track set is a mixed track obtained by mixing multiple tracks. For example, the track set is a mixed track obtained by mixing a vocal track and a track corresponding to at least one instrument. In another example, the track set is a mixed track obtained by superimposing tracks corresponding to at least two instruments. Spectral features include characteristics of how the amplitude information of the track set changes with frequency information. For example, if the audio to be separated is a song and contains five tracks, specifically tracks corresponding to vocals, guitar, bass, electronic synthesizer, and drums, the four track sets obtained after audio separation of the audio to be separated include spectral features corresponding to the vocal track set, guitar track set, drum track set, and mixed track set, respectively. The spectral features corresponding to the mixed track set include spectral features corresponding to the combination of the bass track and the electronic synthesizer track.

[0063] Step 240: Generate audio files corresponding to the n audio track sets according to the spectral features corresponding to the n audio track sets.

[0064] Please refer to Figure 3 , which shows a schematic diagram of an audio separation process. For example, according to actual needs, in some embodiments, the audio to be separated is divided into a vocal track and an accompaniment track.

[0065] Please refer to Figure 4 , which shows a schematic diagram of another audio separation process. In other embodiments, the audio to be separated is divided more finely into a vocal track, a piano track, a bass track, and other instrument tracks. The other instrument tracks contain the sounds of instruments in the audio to be separated other than the vocal track, piano track, and guitar track.

[0066] The audio processing device processes the spectral characteristics of the n audio track sets obtained from audio separation. Using these spectral characteristics and the phase information of the audio to be separated, the audio files corresponding to each of the n audio track sets are obtained, completing the audio separation process. Taking a particular audio track set as an example, the audio processing device processes the spectral characteristics of the track set and the phase information of the audio to be separated to obtain the spectral file corresponding to the track set.

[0067] To sum up, the technical solution provided by the embodiment of the present application obtains the time domain features and texture features of the audio to be separated, and then performs audio separation based on these two features. Since the time domain features and texture features only contain features related to harmonics and do not contain features related to factors such as phase in the audio to be separated, the computational cost of obtaining the time domain features and frequency domain features of the audio to be separated is small during the audio separation process. The time domain features and frequency domain features of the audio to be separated obtained by this method are smaller in dimension than the audio features obtained by directly convolving the audio to be separated. Therefore, the computational cost of audio separation by this method is small and the audio separation speed is fast.

[0068] In addition, by changing the size of parameter n, multiple audio track sets can be obtained, which solves the limitation of related technologies that only vocal tracks and accompaniment tracks can be obtained. For example, the audio separation method provided in this application can extract the audio track sets corresponding to the human voice, string accompaniment, and drum sounds from the audio to be separated. For another example, the audio separation method provided in this application can also separate the audio to be separated of an instrumental ensemble and obtain the audio tracks corresponding to each instrument, meeting the needs of music lovers to obtain a certain type of instrument audio file from the audio to be separated.

[0069] The following describes the process of obtaining the time domain features and texture features of the audio to be separated through two embodiments.

[0070] Please refer to Figure 5 , which shows a schematic diagram of an audio separation method provided by another embodiment of the present application.

[0071] Step 510: Acquire audio to be separated, where the audio to be separated includes at least two audio tracks.

[0072] Step 520: Obtain the time domain features and texture features of the audio to be separated. The time domain features are used to characterize the harmonic correlation of the audio to be separated, and the texture features are used to characterize the harmonic continuity of the audio to be separated.

[0073] In some embodiments, step 520 includes the following sub-steps:

[0074] Step 522: Acquire frequency and amplitude information of the audio to be separated. The frequency and amplitude information is used to represent the frequency and amplitude information of the audio to be separated.

[0075] Optionally, the frequency and amplitude information of the audio to be separated is referred to as a spectrogram of the audio to be separated. In some embodiments, the frequency and amplitude information and phase information of the audio to be separated are obtained by performing a Fourier transform on the audio to be separated. For example, an audio processing device processes the audio to be separated using a short-time Fourier transform to obtain time domain features and texture features corresponding to the audio to be separated.

[0076] As can be seen from the waveform of the music signal, music signals are not stationary. In some cases, signals that differ in the time domain may have very similar spectra. Directly performing a Fourier transform on the audio to be processed will result in distortion. Short-time Fourier transform is used to process the audio to be processed. By windowing, the audio to be separated is segmented in the time domain to obtain several small segments. The signals in these small segments are relatively stationary. The signals in these small frequency bands are Fourier transformed to obtain the frequency-amplitude information of the audio to be separated. Using short-time Fourier transform can avoid distortion of the audio to be separated. Since the time domain information of the audio to be separated contains a large amount of information, and the phase-related information in the time domain information plays a relatively small role in the audio separation process, the frequency-amplitude information of the audio to be separated is obtained by performing a short-time Fourier transform on the audio to be separated or other methods that can separate frequency domain feature information from the time domain information of the audio to be separated. Time domain features and texture features are extracted based on the frequency-amplitude information of the audio to be separated, which helps reduce the amount of computation in the audio separation process and improve the speed of audio separation.

[0077] Step 524: extracting time domain features and texture features based on the frequency information.

[0078] In some embodiments, extracting time domain features and texture features based on amplitude information includes: convolving the amplitude information to obtain amplitude features; dividing the amplitude features to obtain first amplitude features and second amplitude features; wherein the first amplitude features and the second amplitude features are subsets of the amplitude features, and the first amplitude features and the second amplitude features are superimposed to obtain the amplitude features; extracting time domain features based on the first amplitude features; and extracting texture features based on the second amplitude features.

[0079] In some embodiments, an audio separation model is used to separate the audio to be separated and output the separated labeled audio track. The audio separation model is a neural network model with audio separation function, for example, the audio separation model is a neural network such as a recursive neural network, a convolutional neural network, a recurrent neural network, or a combination thereof.

[0080] Optionally, the audio separation model includes an amplitude coding network, a time domain extraction network, and a texture extraction network. The amplitude coding network is used to perform feature sorting on the amplitude information to obtain the amplitude features of the audio to be separated. For example, the amplitude coding network is used to perform convolution on the amplitude information to obtain the amplitude features. The time domain extraction network is used to extract time domain features. For example, the time domain extraction network is used to extract time domain features based on the first amplitude features. The texture extraction network is used to extract texture features. For example, the texture extraction network is used to extract texture features based on the second amplitude features.

[0081] Frequency-amplitude features refer to a type of feature information related to frequency and amplitude extracted from the frequency-amplitude information of the audio to be separated. The frequency-amplitude coding network in the audio separation model uses convolution to perform feature sorting on the frequency-amplitude information of the audio to be separated, and extracts frequency-amplitude features from the frequency-amplitude information of the audio to be separated. In some embodiments, a larger convolution kernel is used in the frequency-amplitude coding network to extract features from the frequency-amplitude information of the audio to be separated. For example, the audio separation model inputs the frequency-amplitude information of the audio to be separated into the frequency-amplitude coding network. The frequency-amplitude coding network includes three convolution layers. Each convolution layer uses a convolution kernel of size 7*7 to convolve the feature information input to the convolution layer. The output of the last convolution layer is the frequency-amplitude feature. Optionally, the convolution kernel size in the frequency-amplitude coding network is greater than or equal to 3*3. In the frequency-amplitude coding network, the number of convolution layers and the size of the convolution kernel are set according to actual needs and are not limited here.

[0082] Using large-size convolution kernels for convolution in the convolutional layer of the frequency-amplitude coding network can abstract the input frequency-amplitude information into spectral features of multiple dimensions, which is beneficial to increasing the receptive field of the convolution process, reducing the coupling of frequency-amplitude features, and helping the subsequent network to better learn the specific features of the audio to be separated from the frequency-amplitude features.

[0083] The audio separation model divides the amplitude features extracted by the amplitude coding network, and obtains the first amplitude feature and the second amplitude feature by dividing. For example, a certain amplitude feature is a matrix with three dimensions: width, time, and number of channels. The number of channels of the amplitude feature is 64, and the corresponding time and width on each channel are equal. The audio separation model uses the first 32 channels of the amplitude feature as the first amplitude feature and the last 32 channels as the second amplitude feature. In some embodiments, in order to ensure that the accuracy of the audio track set obtained after audio separation is higher, the amplitude coding network of the audio separation model is structurally improved so that the number of channels of the amplitude feature extracted by the amplitude coding network is larger, that is, the number of channels of the first amplitude feature and the second amplitude feature obtained after separation is larger. The more data contained in the first amplitude feature and the second amplitude feature, the better the accuracy of the result separated by the audio separation model. However, increasing the number of channels of the amplitude feature will increase the computational complexity of the audio separation model and slow down the audio separation speed. The number of channels of the frequency amplitude feature, the number of channels of the first frequency amplitude feature, and the number of channels of the second frequency amplitude feature can be comprehensively determined based on requirements such as the accuracy of audio separation and the speed of audio separation, and this application does not limit this.

[0084] The audio separation model inputs the first frequency-amplitude feature into the time-domain extraction network, which extracts the time-domain features of the audio to be separated from the first frequency-amplitude feature. In some embodiments, the audio processing device uses a recursive neural network as the time-domain extraction network, such as a Bi-directional Long Short-Term Memory (BilSTM) neural network or a Bi-directional Gated Recurrent Unit (BiGRU) neural network.

[0085] The audio separation model inputs the second frequency-amplitude feature into the texture extraction network, which extracts the texture features of the audio to be separated from the second frequency-amplitude feature. In some embodiments, the audio processing device uses a convolutional neural network as a time-domain extraction network. For example, a convolutional neural network with a convolution kernel size of 3*3 in each convolution layer convolves the second frequency-amplitude feature to obtain the texture features of the audio to be separated. The number of convolution layers and convolution kernel size of the convolutional neural network are set based on actual conditions such as the computing power of the device and are not limited here.

[0086] In other embodiments, time domain features and texture features are extracted based on frequency amplitude information, including: convolving the frequency amplitude information to obtain a third frequency amplitude feature, and extracting time domain features based on the frequency amplitude feature; convolving the frequency amplitude information to obtain a fourth frequency amplitude feature, and extracting texture features based on the frequency amplitude feature.

[0087] Optionally, the audio separation model includes a combined time-domain extraction network and a combined texture extraction network. The audio processing device inputs frequency and amplitude information of the audio to be separated into the audio separation model. The combined time-domain extraction network performs convolution processing on the frequency and amplitude information and extracts the time-domain features of the audio to be separated; the combined texture extraction network performs convolution processing on the frequency and amplitude information and extracts the texture features of the audio to be separated.

[0088] Step 530: Obtain spectral features corresponding to n audio track sets based on the time domain features and texture features. The spectral features are used to characterize the frequency and amplitude information of the audio track sets. Each audio track set includes one audio track or a combination of multiple audio tracks in the audio to be separated. n is a positive integer.

[0089] In some embodiments, the audio separation model further includes a track feature extraction network. Step 530 includes the following sub-steps:

[0090] In step 532, the time domain features and the texture features are fused to obtain mixed features. Fusion processing refers to unifying the dimensions between the time domain features and the texture features, and adding the features of the corresponding dimensions in the unified time domain features and texture features.

[0091] Step 534: Process the mixed features through the audio track feature extraction network to generate spectral features corresponding to the n audio track sets.

[0092] In some embodiments, the dimension of the time domain feature extracted based on the first amplitude feature is the same as the dimension of the second amplitude feature. In the process of fusing the time domain feature and the texture feature, the hybrid feature can be obtained by simply adding the feature values ​​at corresponding positions of the time domain feature and the texture feature; optionally, the time domain feature, the texture feature, and the hybrid feature have the same dimension.

[0093] In some embodiments, the dimensions of the time domain features extracted based on the first amplitude feature and the dimensions of the texture features extracted based on the second amplitude feature are not equal. Before the time domain features and texture features are fused, the time domain features and texture features need to be dimensionally matched so that the dimensions of the time domain adjustment and the texture are equal. In some embodiments, when the time domain extraction network uses a recurrent neural network, the recurrent neural network performs dimensionality reduction before outputting the time domain features, resulting in a smaller number of channels in the time domain features than in the texture features. Before the time domain features and texture features are fused, the time domain features need to be replicated in the channel dimension so that the number of channels in the time domain features and the texture are equal. For example, the number of channels in the time domain features is 1 and the number of channels in the texture features is 2. The audio separation model replicates the time domain features to obtain replicated time domain features, and uses the replicated time domain features to expand the number of channels in the time domain features so that the number of channels in the time domain features becomes 2, the same as the number of channels in the texture features. The data at corresponding positions of the time domain features and texture features of the same dimensions are added together to obtain a mixed feature.

[0094] The spectral features of a track set include the frequencies of the track set and the corresponding amplitude information. Optionally, the spectral features of a track set include the frequency-amplitude information of the track set. This frequency-amplitude information is referred to as the spectrogram of the track set. The track feature extraction network convolves the mixed features to extract the spectral features of the track set. In some embodiments, a fully convolutional network, such as a U-Net (U-shaped network) fully convolutional neural network, is used as the track feature extraction network in the audio separation model.

[0095] Please refer to Figure 6 , which shows the network structure of an audio separation model. The audio separation model includes a frequency-amplitude encoding network, a time-domain extraction network, a texture extraction network, and an audio track acquisition network. The types and specific functions of each network are described above and will not be detailed here.

[0096] Please refer to Figure 7, which shows another network structure of the audio separation model, which includes a combined time domain extraction network, a combined texture extraction network and an audio track acquisition network. The combined time domain extraction network has the capabilities of both the frequency amplitude coding network and the time domain extraction network, and the combined texture extraction network has the capabilities of both the frequency amplitude coding network and the texture extraction network.

[0097] Step 540: Generate audio files corresponding to the n audio track sets according to the spectral features corresponding to the n audio track sets.

[0098] In some embodiments, audio files corresponding to n audio track sets are generated based on the spectral features corresponding to the n audio track sets, including: obtaining phase information of the audio to be separated, where the phase information is used to characterize the phase of the audio to be separated; performing an inverse Fourier transform on the spectral features corresponding to the audio track sets based on the phase information to generate audio files corresponding to the audio track sets.

[0099] After the audio processing device performs a short-time Fourier transform on the audio to be separated to generate frequency-amplitude information, it can obtain phase information of the audio to be separated based on the frequency-amplitude information. The audio processing device performs an inverse Fourier transform on the spectral characteristics corresponding to the n audio track sets and the phase information, generating n audio files, and outputs the n audio files separately.

[0100] Please refer to Figure 8 , which shows a schematic diagram of an audio separation method provided by an embodiment of the present application.

[0101] After obtaining the audio to be separated, the audio processing device performs a short-time Fourier transform on it to obtain the frequency and amplitude information of the audio to be separated. The audio processing device inputs this frequency and amplitude information into the audio separation model. The feature encoding network uses a large-scale convolution to perform feature analysis on this frequency and amplitude information, obtaining high-level features in the frequency and amplitude information, namely the frequency and amplitude features of the audio sample to be separated. The audio separation model divides the frequency-amplitude features into the first frequency-amplitude features and the second frequency-amplitude features; wherein the first frequency-amplitude features and the second frequency-amplitude features are subsets of the frequency-amplitude features; the first frequency-amplitude features are extracted through the time domain extraction network to obtain the time domain features; the second frequency-amplitude features are extracted through the texture extraction network to obtain the texture features; the audio separation model performs dimension matching on the time domain features and the texture features, and performs fusion processing to obtain the mixed features, the track feature generation network convolves the mixed features, and finally outputs the spectral features of track set 1 and the spectral features of track set 2. The audio file corresponding to track set 1 is obtained by performing an inverse Fourier transform on the spectral features of track set 1 and the phase information of the audio to be separated, and the audio file corresponding to track set 2 is obtained by performing an inverse Fourier transform on the phase information of the spectral features of track set 2 and the audio to be separated.

[0102] In actual application, when the user only needs to separate a specific audio track from the audio to be separated, the audio separation model can optionally first separate the audio to be separated to obtain n audio track sets (n is greater than or equal to 1), and then select the audio track required by the user for output. This method can ensure that the user obtains an audio file corresponding to a specific audio track with better quality. Optionally, after the audio separation model separates the audio to be separated, it only generates the audio track required by the user. This method can reduce the amount of calculation in the audio separation process, speed up the separation speed of the audio to be separated, and specifically separate a type of audio track from the audio to be separated.

[0103] Below, the training process of the audio separation model is introduced and explained through an embodiment. The contents involved in the use process of the audio separation model and the contents involved in the training process correspond to each other, and the two are interconnected. If there is no detailed explanation on one side, you can refer to the description on the other side.

[0104] Please refer to Figure 9 , which shows a flow chart of a method for training an audio separation model provided by an embodiment of the present application. The execution subject of each step of the method is a model training device 10 in the implementation environment. Below, the model training device 10 is used as the execution subject. The method may include at least one of the following steps (910-940):

[0105] Step 910: Acquire training data for the audio separation model. The training data includes an audio sample to be separated and n labeled audio tracks corresponding to the audio sample to be separated. The audio sample to be separated includes at least two audio tracks, and n is a positive integer.

[0106] In step 920 , the time domain features and texture features of the audio samples to be separated are obtained through the audio separation model. The time domain features are used to characterize the harmonic correlation of the audio samples to be separated, and the texture features are used to characterize the harmonic continuity of the audio samples to be separated.

[0107] Step 930: Obtain spectral features corresponding to n track sets based on the time domain features and texture features. The spectral features are used to characterize the frequency and amplitude information of the track sets. Each track set includes one track or a combination of multiple tracks in the audio sample to be separated.

[0108] Step 940 : Calculate the training loss of the audio separation model based on the spectral features corresponding to the n audio track sets and the spectral features corresponding to the n labeled audio tracks, and train the audio separation model based on the training loss.

[0109] In some embodiments, obtaining training data for an audio separation model includes: obtaining an audio data set, the audio data set including multiple source audio track audios; selecting m source audio track audios from the multiple source audio track audios, where m is a positive integer greater than or equal to n; performing mixing processing on the m source audio track audios to obtain audio samples to be separated; and generating n label audio tracks corresponding to the audio samples to be separated based on the m source audio track audios. Source audio track audio refers to an audio file obtained by recording, electronic synthesis, etc. The source audio track audio can be obtained from an audio data set, and the source and type of the source audio track audio are not limited here. Mixing processing refers to the operation of mixing m source audio tracks to obtain mixed audio. In some embodiments, the model training device 10 aligns the time axes of the m source audio track audios, plays them uniformly, completes the mixing processing, and obtains the audio samples to be separated.

[0110] Label tracks refer to the types of tracks that the audio separation model can separate from the audio to be separated. After training, the audio separation model has the ability to separate n label tracks from the audio to be separated.

[0111] If the playback durations of the m source audio tracks are different, the shorter source audio tracks are optionally played repeatedly to extend the playback duration, while the longer source audio tracks are clipped to shorten their playback duration. The m source audio tracks with equal playback durations are then mixed to obtain the audio samples to be separated.

[0112] When m is equal to n, each label audio track has a corresponding source audio track audio. For example, the model training device 10 obtains two source audio track audios from the audio data set, namely the source audio track audio source corresponding to the human voice and the source audio track audio source corresponding to the guitar sound. The audio separation model includes two label audio tracks, namely the track set corresponding to the human voice and the track set corresponding to the guitar sound. The track set corresponding to the human voice can be directly obtained based on the track set audio source corresponding to the human voice; the track set corresponding to the guitar sound can be directly obtained from the track set audio source corresponding to the guitar sound. When m is greater than n, there are some label tracks that are obtained by mixing multiple source track audios. For example, the model training device 10 obtains 5 source track audios from the audio data set, which are the source track audios corresponding to piano, guitar, vocals, drums and triangle respectively. The audio separation model includes 4 label tracks, which are the label tracks corresponding to piano, guitar, vocals and percussion respectively. The label tracks corresponding to piano, guitar and vocals can be directly determined from the corresponding source track audios respectively. The label track corresponding to percussion needs to be determined by mixing the source track audio corresponding to drums and the source track audio corresponding to triangle, and then determined according to the mixed track audio.

[0113] In some embodiments, the audio separation model includes a frequency-amplitude coding network, a time domain extraction network, and a texture extraction network; the time domain features and texture features of the audio samples to be separated are obtained through the audio separation model, including: obtaining the frequency-amplitude information of the audio samples to be separated, the frequency-amplitude information is used to characterize the frequency and amplitude information of the audio samples to be separated; convolving the frequency-amplitude information through the frequency-amplitude coding network to obtain frequency-amplitude features; dividing the frequency-amplitude features to obtain first frequency-amplitude features and second frequency-amplitude features; wherein the first frequency-amplitude features and the second frequency-amplitude features are subsets of the frequency-amplitude features, and the first frequency-amplitude features and the second frequency-amplitude features can be superimposed to obtain the frequency-amplitude features; extracting the time domain features based on the first frequency-amplitude features through the time domain extraction network; extracting the texture features based on the second frequency-amplitude features through the texture extraction network.

[0114] For the detailed process of obtaining the time domain features and texture features of the audio sample to be separated through the audio separation model, please refer to the previous embodiment and will not be repeated here.

[0115] In some embodiments, the audio separation model also includes: a track feature extraction network, which obtains spectral features corresponding to n track sets based on texture features and time domain features, including: fusing the time domain features and texture features to obtain mixed features; wherein the fusion process refers to unifying the dimensions between the time domain features and the texture features, and adding the features of the corresponding dimensions in the unified time domain features and texture features; processing the mixed features through the track feature extraction network to generate spectral features corresponding to n track sets.

[0116] The detailed process of fusing the time domain features and the texture features to obtain the mixed features can be found in the previous embodiment and will not be described in detail here.

[0117] In some embodiments, calculating the training loss of the audio separation model based on the spectral features corresponding to the n audio track sets and the spectral features corresponding to the n labeled audio tracks includes:

[0118] For each of the n track sets, the degree of distinction between the spectral features of the track set and the spectral features of the label track corresponding to the track set is calculated to obtain n distinctions; and the training loss of the audio separation model is determined based on the n distinctions.

[0119] The degree of distinction between the spectral features of a track set and the tagged tracks corresponding to the track set is used to characterize the degree of distinction between the track set and the corresponding tagged tracks. In some embodiments, the spectral features of the track set and the spectral features of the tagged tracks have the same dimensions. The degree of distinction between the spectral features of a track set and the spectral features of the tagged tracks corresponding to the track set is obtained by calculating the absolute value of the data difference at corresponding positions in the two spectral features and calculating the average.

[0120] The degree of distinction between the spectral features of the track set and the spectral features of the labeled tracks corresponding to the track set can be calculated by other distance calculation methods, such as calculating the sum of the absolute values ​​of the differences between the spectral features of the track set and the spectral features of the labeled tracks corresponding to the track set. The method for calculating the degree of distinction is not limited here.

[0121] The audio separation model determines a loss of the audio separation model based on the n distinctnesses, including calculating an average of the n distinctnesses to determine the loss of the audio separation model, or calculating a sum of the n distinctnesses to determine the loss of the audio separation model. After determining the loss of the audio separation model, the computer device adjusts network parameters of various components of the audio separation model. In some embodiments, the computer device adjusts the parameters of the audio separation model using a gradient descent method.

[0122] After the loss of the audio separation model converges to the target value, the training of the audio separation model is completed.

[0123] In summary, the training data of the audio separation model is obtained, and the time domain features and texture features of the audio samples to be separated are obtained from the audio separation model; the spectral features corresponding to the n audio track sets are obtained according to the time domain features and texture features, and the training loss of the audio separation model is calculated according to the spectral features corresponding to the n audio track sets and the spectral features corresponding to the n labeled audio tracks, and the audio separation model is trained based on the training loss so that the trained audio separation model has the ability to generate n labeled audio tracks. The spectral features of the n audio track sets are obtained using the time domain features and texture features of the audio to be separated, and the amount of calculation in the audio separation process is small and the audio separation speed is fast. In addition, during the training of the audio separation model, the more types of labeled audio tracks there are, the stronger the final separation ability of the audio separation model will be, and the quality of the audio track set obtained through audio separation will be better. The larger N is, the more factors that can affect the training loss of the audio separation model during training. For example, the labeled audio tracks of an audio separation model include: piano labeled audio tracks, guitar labeled audio tracks, bass labeled audio tracks, and vocal labeled audio tracks. During the training process of this audio separation model, the parameters in the model responsible for separating the piano labeled audio tracks, guitar labeled audio tracks, bass labeled audio tracks, and vocal labeled audio tracks influence and restrict each other, realizing transfer learning and improving the effectiveness of model training. This ensures that the audio separation model has good separation effect and the quality of the audio track set obtained through separation is better.

[0124] Please refer to Figure 10 , which shows a schematic diagram of the audio separation model training process of this application.

[0125] After the model training device 10 obtains the training data, it performs short-time Fourier transform on the audio samples to be separated in the training data to obtain the frequency and amplitude information of the audio samples to be separated, and inputs the frequency and amplitude information into the audio separation model. The feature encoding network uses a larger convolution kernel to perform feature sorting on the frequency and amplitude information to obtain high-level features in the frequency and amplitude information, that is, the frequency and amplitude features of the audio samples to be separated. The audio separation model divides the frequency and amplitude features into a first frequency and amplitude feature and a second frequency and amplitude feature; the first frequency and amplitude features are extracted through a time domain extraction network to obtain time domain features; the second frequency and amplitude features are extracted through a texture extraction network to obtain texture features; the audio separation model performs dimension matching on the time domain features and texture features, and performs fusion processing to obtain mixed features. The track feature generation network convolves the mixed features and finally outputs the spectral features of the track set. Optionally, if the audio separation model has n labeled tracks, the track feature generation network finally outputs n spectral features, and the n spectral features correspond to n track sets respectively. Figure 10 As shown in the figure, the track separation network outputs the spectral features corresponding to track set 1, track set 2, and track set 3, and calculates the distinction between the spectral features of the three track sets and the frequency amplitude features of the corresponding labeled tracks to obtain the training loss of the audio separation model; based on the training loss, the parameters in the audio separation model are adjusted, and the above steps are repeated until the training loss of the audio separation model converges to the target value, completing the training of the audio separation model.

[0126] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0127] Please refer to Figure 11 , which shows a block diagram of an audio separation device provided by one embodiment of the present application. The device has the function of implementing the above-mentioned audio separation method, which can be implemented by hardware or by hardware executing corresponding software. The device can be the audio processing device described above, or it can be set in an audio processing device. The device 1100 may include: an audio acquisition module 1110, a feature extraction module 1120, a spectrum generation module 1130, and a track generation module 1140.

[0128] The audio acquisition module 1110 is configured to acquire audio to be separated, where the audio to be separated includes at least two audio tracks.

[0129] The feature extraction module 1120 is used to obtain the time domain features and texture features of the audio to be separated, wherein the time domain features are used to characterize the harmonic correlation of the audio to be separated, and the texture features are used to characterize the harmonic continuity of the audio to be separated.

[0130] The spectrum generation module 1130 is used to obtain spectrum features corresponding to n audio track sets based on the time domain features and the texture features. The spectrum features are used to characterize the frequency and amplitude information of the audio track sets. Each audio track set includes one audio track or a combination of multiple audio tracks in the audio to be separated, and n is a positive integer.

[0131] The audio track generation module 1140 is configured to generate audio files corresponding to the n audio track sets according to the spectral features corresponding to the n audio track sets.

[0132] In some embodiments, the feature extraction module 1120 includes: a frequency information acquisition submodule and a feature extraction submodule.

[0133] The frequency-amplitude information acquisition submodule is used to acquire the frequency-amplitude information of the audio to be separated, where the frequency-amplitude information is used to represent the frequency and amplitude information of the audio to be separated.

[0134] The feature extraction submodule is used to extract the time domain features and the texture features based on the frequency information.

[0135] In some embodiments, the feature extraction submodule is used to convolve the frequency information to obtain frequency features; divide the frequency features to obtain a first frequency feature and a second frequency feature; wherein the first frequency feature and the second frequency feature are subsets of the frequency features, and the first frequency feature and the second frequency feature are superimposed to obtain the frequency feature; extract the time domain feature based on the first frequency feature; and extract the texture feature based on the second frequency feature.

[0136] In some embodiments, the audio separation model includes a frequency amplitude coding network, a time domain extraction network and a texture extraction network; wherein the frequency amplitude coding network is used to convolve the frequency amplitude information to obtain the frequency amplitude feature; the time domain extraction network is used to extract the time domain feature based on the first frequency amplitude feature; and the texture extraction network is used to extract the texture feature based on the second frequency amplitude feature.

[0137] In some embodiments, the audio separation model further includes a track feature extraction network. A spectrum generation module 1130 is configured to fuse the time domain features and the texture features to obtain mixed features. The fusion process includes unifying the dimensions of the time domain features and the texture features and adding the features of the corresponding dimensions of the unified time domain features and the texture features. The mixed features are processed by the track feature extraction network to generate spectrum features corresponding to the n track sets.

[0138] In some embodiments, the audio track generation module 1140 is used to obtain phase information of the audio to be separated, where the phase information is used to characterize the phase of the audio to be separated; perform an inverse Fourier transform on the spectral features corresponding to the audio track set according to the phase information to generate an audio file corresponding to the audio track set.

[0139] Please refer to Figure 12 , which shows a block diagram of a training device for an audio separation model provided by an embodiment of the present application. The device has the function of implementing the training method of the above-mentioned audio separation model, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be the model training device 10 introduced above, or it can be set in the model training device 10. The device 1200 may include: a data acquisition module 1210, a feature extraction module 1220, a spectrum generation module 1230 and a model training module 1240.

[0140] The data acquisition module 1210 is used to obtain training data for the audio separation model. The training data includes an audio sample to be separated and n labeled audio tracks corresponding to the audio sample to be separated. The audio sample to be separated includes at least two audio tracks, and n is a positive integer.

[0141] The feature extraction module 1220 is used to obtain the time domain features and texture features of the audio samples to be separated through the audio separation model, wherein the time domain features are used to characterize the harmonic correlation of the audio samples to be separated, and the texture features are used to characterize the harmonic continuity of the audio samples to be separated.

[0142] The spectrum generation module 1230 is used to obtain spectrum features corresponding to n audio track sets based on the time domain features and the texture features. The spectrum features are used to characterize the frequency and amplitude information of the audio track sets. Each audio track set includes one audio track or a combination of multiple audio tracks in the audio sample to be separated.

[0143] The model training module 1240 is used to calculate the training loss of the audio separation model based on the spectral features corresponding to the n audio track sets and the spectral features corresponding to the n labeled audio tracks, and train the audio separation model based on the training loss.

[0144] In some embodiments, the data acquisition module 1210 is configured to select m source audio tracks from the multiple source audio tracks, where m is a positive integer greater than or equal to n; perform mixing processing on the m source audio tracks to obtain the audio samples to be separated; and generate n label audio tracks corresponding to the audio samples to be separated based on the m source audio tracks.

[0145] In some embodiments, the audio separation model includes a frequency-amplitude coding network, a time-domain extraction network, and a texture extraction network. The feature extraction module 1220 is used to obtain the frequency-amplitude information of the audio sample to be separated, and the frequency-amplitude information is used to characterize the frequency and amplitude information of the audio sample to be separated; convolve the frequency-amplitude information through the frequency-amplitude coding network to obtain frequency-amplitude features; divide the frequency-amplitude features to obtain first frequency-amplitude features and second frequency-amplitude features; wherein the first frequency-amplitude features and the second frequency-amplitude features are subsets of the frequency-amplitude features, and the first frequency-amplitude features and the second frequency-amplitude features are superimposed to obtain the frequency-amplitude features; extract the time-domain features based on the first frequency-amplitude features through the time-domain extraction network; and extract the texture features based on the second frequency-amplitude features through the texture extraction network.

[0146] In some embodiments, the spectrum generation module 1230 is used to fuse the time domain features and the texture features to obtain mixed features; wherein the fusion processing refers to unifying the dimensions between the time domain features and the texture features, and adding the features of the corresponding dimensions of the unified time domain features and the texture features; the mixed features are processed by the audio track feature extraction network to generate spectrum features corresponding to the n audio track sets.

[0147] In some embodiments, the model training module 1240 is used to calculate, for each of the n track sets, the degree of distinction between the spectral features of the track set and the spectral features of the label track corresponding to the track set, to obtain n degrees of distinction; and determine the training loss of the audio separation model based on the n degrees of distinction.

[0148] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0149] Please refer to Figure 13 , which shows a schematic diagram of a computer device provided by an embodiment of the present application. The computer device 1300 may be Figure 1 The audio processing device 20 in the implementation environment shown is used to implement the above audio separation method; it can also be Figure 1 The model training device 10 in the illustrated implementation environment is used to implement the above-mentioned audio separation model training method.

[0150] Typically, the computer device 1300 includes a processor 1301 and a memory 1302 .

[0151] The processor 1301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1301 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0152] The memory 1302 may include one or more computer-readable storage media, which may be non-transitory, and may also include high-speed random access memory and non-volatile memory, such as one or more magnetic disk storage devices and flash memory storage devices.

[0153] Those skilled in the art will understand that Figure 13 The structure shown in the figure does not constitute a limitation on the device 1300, and the device 1300 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0154] In some embodiments, a computer program is stored in a memory of a computer device, and the computer program is loaded and executed by a processor to implement the audio separation method or the audio separation model training method as described above.

[0155] The present application also provides a computer-readable storage medium, which stores a computer program. The computer program is loaded and executed by a processor to implement the audio separation method or audio separation model training method as described above.

[0156] Optionally, the computer storage medium includes RAM, ROM, flash memory or other solid-state storage technology, other optical storage such as CD-ROM, tape cassettes, magnetic tape, disk storage, etc.

[0157] The present application also provides a computer program product or computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor reads the computer instructions from the computer-readable storage medium to implement the audio separation method or audio separation model training method provided in the above-mentioned method embodiments.

[0158] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application do not limit this.

[0159] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An audio separation method, characterized in that: The method comprises: Acquire audio to be separated, where the audio to be separated includes at least two audio tracks; Performing Fourier transform on the audio to be separated to obtain frequency and amplitude information and phase information of the audio to be separated, wherein the frequency and amplitude information is used to represent the frequency and amplitude information of the audio to be separated, and the phase information is used to represent the phase of the audio to be separated; Extracting time domain features and texture features of the audio to be separated based on the frequency amplitude information, wherein the time domain features are used to characterize the harmonic correlation of the audio to be separated, and the texture features are used to characterize the harmonic continuity of the audio to be separated; Obtaining, based on the time domain features and the texture features, spectral features corresponding to n audio track sets, respectively, wherein the spectral features are used to characterize frequency and amplitude information of the audio track sets, each audio track set including one audio track or a combination of multiple audio tracks of the audio to be separated, where n is a positive integer; Audio files corresponding to the n audio track sets are generated according to the frequency spectrum features corresponding to the n audio track sets and the phase information of the audio to be separated.

2. The method according to claim 1, characterized in that The extracting the time domain features and texture features of the audio to be separated based on the frequency amplitude information includes: Convolving the frequency and amplitude information to obtain frequency and amplitude features; Dividing the frequency amplitude feature to obtain a first frequency amplitude feature and a second frequency amplitude feature; wherein the first frequency amplitude feature and the second frequency amplitude feature are subsets of the frequency amplitude feature, and the first frequency amplitude feature and the second frequency amplitude feature are superimposed to obtain the frequency amplitude feature; Extracting the time domain feature based on the first frequency amplitude feature; The texture feature is extracted based on the second amplitude feature.

3. The method according to claim 2, characterized in that The audio separation model includes a frequency-amplitude coding network, a time-domain extraction network, and a texture extraction network; among them, The frequency amplitude coding network is used to perform convolution on the frequency amplitude information to obtain the frequency amplitude feature; The time domain extraction network is used to extract the time domain feature based on the first frequency amplitude feature; The texture extraction network is used to extract the texture feature based on the second amplitude feature.

4. The method according to claim 3, characterized in that The audio separation model further includes: a track feature extraction network, which obtains spectral features corresponding to n track sets based on the time domain features and the texture features, including: Performing a fusion process on the time domain features and the texture features to obtain a mixed feature; wherein the fusion process refers to unifying the dimensions between the time domain features and the texture features, and adding the features of the corresponding dimensions in the unified time domain features and the texture features; The mixed features are processed by the audio track feature extraction network to generate spectral features corresponding to the n audio track sets.

5. The method according to claim 1, wherein Generating audio files corresponding to the n audio track sets respectively according to the spectral features corresponding to the n audio track sets and the phase information of the audio to be separated includes: Perform an inverse Fourier transform on the frequency spectrum features corresponding to the audio track set according to the phase information to generate an audio file corresponding to the audio track set.

6. A training method for an audio separation model, characterized in that: The method comprises: Obtaining training data for the audio separation model, the training data including an audio sample to be separated and n labeled audio tracks corresponding to the audio sample to be separated, the audio sample to be separated including at least two audio tracks, where n is a positive integer; Performing Fourier transform on the audio sample to be separated using the audio separation model to obtain frequency and amplitude information of the audio sample to be separated, where the frequency and amplitude information is used to represent the frequency and amplitude information of the audio sample to be separated, and the phase information is used to represent the phase of the audio sample to be separated; Extracting time domain features and texture features of the audio samples to be separated based on the frequency and amplitude information using the audio separation model, wherein the time domain features are used to characterize the harmonic correlation of the audio samples to be separated, and the texture features are used to characterize the harmonic continuity of the audio samples to be separated; Obtaining, based on the time domain features and the texture features, spectral features corresponding to n audio track sets, respectively, wherein the spectral features are used to characterize frequency and amplitude information of the audio track sets, each audio track set including one audio track or a combination of multiple audio tracks of the audio sample to be separated; The training loss of the audio separation model is calculated according to the spectral features corresponding to the n audio track sets and the spectral features corresponding to the n labeled audio tracks, and the audio separation model is trained based on the training loss.

7. The method according to claim 6, characterized in that The obtaining of training data for the audio separation model includes: Acquire an audio data set, wherein the audio data set includes multiple source audio tracks; Select m source audio tracks from the plurality of source audio tracks, where m is a positive integer greater than or equal to n; Mixing the m source audio tracks to obtain the audio samples to be separated; Based on the m source audio tracks, n label audio tracks corresponding to the audio samples to be separated are generated.

8. The method according to claim 6, characterized in that The audio separation model includes a frequency-amplitude coding network, a time-domain extraction network, and a texture extraction network; and extracting the time-domain features and texture features of the audio sample to be separated based on the frequency-amplitude information by the audio separation model includes: Convolving the frequency information through the frequency coding network to obtain frequency features; Dividing the frequency amplitude feature to obtain a first frequency amplitude feature and a second frequency amplitude feature; wherein the first frequency amplitude feature and the second frequency amplitude feature are subsets of the frequency amplitude feature, and the first frequency amplitude feature and the second frequency amplitude feature are superimposed to obtain the frequency amplitude feature; Extracting the time domain feature based on the first frequency amplitude feature through the time domain extraction network; The texture feature is extracted based on the second amplitude feature through the texture extraction network.

9. The method according to claim 8, characterized in that The audio separation model further includes: a track feature extraction network, which obtains spectral features corresponding to n track sets based on the texture features and the time domain features, including: Performing a fusion process on the time domain features and the texture features to obtain a mixed feature; wherein the fusion process refers to unifying the dimensions between the time domain features and the texture features, and adding the features of the corresponding dimensions in the unified time domain features and the texture features; The mixed features are processed by the audio track feature extraction network to generate spectral features corresponding to the n audio track sets.

10. The method according to claim 6, characterized in that Calculating the training loss of the audio separation model according to the spectral features corresponding to the n audio track sets and the spectral features corresponding to the n labeled audio tracks includes: For each of the n track sets, calculating the degree of distinction between the spectral features of the track set and the spectral features of the label track corresponding to the track set, to obtain n degrees of distinction; A training loss of the audio separation model is determined based on the n distinctivenesses.

11. An audio separation device, characterized in that: The device comprises: An audio acquisition module, configured to acquire audio to be separated, wherein the audio to be separated includes at least two audio tracks; a feature extraction module, configured to perform Fourier transform on the audio to be separated to obtain frequency and amplitude information and phase information of the audio to be separated, wherein the frequency and amplitude information is used to characterize the frequency and amplitude information of the audio to be separated, and the phase information is used to characterize the phase of the audio to be separated; and extract time domain features and texture features of the audio to be separated based on the frequency and amplitude information, wherein the time domain features are used to characterize the harmonic correlation of the audio to be separated, and the texture features are used to characterize the harmonic continuity of the audio to be separated; a spectrum generation module, configured to obtain spectrum features corresponding to n audio track sets based on the time domain features and the texture features, wherein the spectrum features are used to characterize frequency and amplitude information of the audio track sets, each audio track set including one audio track or a combination of multiple audio tracks of the audio to be separated, where n is a positive integer; The audio track generation module is used to generate audio files corresponding to the n audio track sets respectively according to the spectral features corresponding to the n audio track sets and the phase information of the audio to be separated.

12. A training device for an audio separation model, characterized in that: The device comprises: A data acquisition module, configured to acquire training data for the audio separation model, wherein the training data includes an audio sample to be separated and n labeled audio tracks corresponding to the audio sample to be separated, wherein the audio sample to be separated includes at least two audio tracks, and n is a positive integer; a feature extraction module, configured to perform Fourier transform on the audio sample to be separated to obtain frequency and amplitude information of the audio sample to be separated, wherein the frequency and amplitude information is used to characterize the frequency and amplitude information of the audio sample to be separated, and the phase information is used to characterize the phase of the audio sample to be separated; and extract time domain features and texture features of the audio sample to be separated based on the frequency and amplitude information through the audio separation model, wherein the time domain features are used to characterize the harmonic correlation of the audio sample to be separated, and the texture features are used to characterize the harmonic continuity of the audio sample to be separated; a spectrum generation module, configured to obtain spectrum features corresponding to n audio track sets respectively based on the time domain features and the texture features, wherein the spectrum features are used to characterize frequency and amplitude information of the audio track sets, each audio track set including one audio track or a combination of multiple audio tracks of the audio sample to be separated; A model training module is used to calculate the training loss of the audio separation model based on the spectral features corresponding to the n audio track sets and the spectral features corresponding to the n labeled audio tracks, and train the audio separation model based on the training loss.

13. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the audio separation method according to any one of claims 1 to 5, or to implement the training method of the audio separation model according to any one of claims 6 to 10.

14. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which is loaded and executed by a processor to implement the audio separation method according to any one of claims 1 to 5, or the training method of the audio separation model according to any one of claims 6 to 10.

15. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. The processor reads and executes the computer instructions from the computer-readable storage medium to implement the audio separation method according to any one of claims 1 to 5, or the training method of the audio separation model according to any one of claims 6 to 10.

Images