Training method and device for audio track separation model and audio track separation method and device

Abstract

The present disclosure provides a method and device for training a track separation model and a method and device for track separation. The training method may include: obtaining a mixed audio signal and a pure target track signal, wherein the mixed audio signal includes a pure target track signal; using an encoder to extract features of the mixed audio signal to obtain time-frequency features of the mixed audio signal; based on the time-frequency features, determining the scale information of at least one attention unit; using at least one attention unit in a separator, separating and processing the time-frequency features based on the scale information of each attention unit to obtain target track information; using a decoder to decode the target track information to obtain an estimated target track signal; training a track separation model based on the pure target track signal and the estimated target track signal to obtain a trained track separation model. The present disclosure can separate audio signals more accurately.

Description

Technical Field

[0001] The present disclosure relates to the technical field of audio processing, and in particular to a method and device for training a track separation model, a track separation method and device, an electronic device, a storage medium, and a program product. Background Art

[0002] Track separation technology can separate vocals, drums, bass and other instrument signals from the mixed audio signal. The separated single-track signal can be used for secondary creation of music, such as replacing the original vocals in the music with the user's own singing voice, or re-editing the separated instrument signal to generate a mixed result of other styles. However, the separation results obtained using traditional signal processing methods are obviously distorted. Summary of the invention

[0003] The present disclosure provides a method and device for training a track separation model, a track separation method and device, an electronic device and a storage medium to at least solve the above-mentioned problems.

[0004] According to a first aspect of an embodiment of the present disclosure, a training method for a track separation model is provided, the track separation model comprising an encoder, a separator and a decoder, the separator comprising at least one attention unit, the method may include: obtaining a mixed audio signal and a pure target track signal, wherein the mixed audio signal comprises the pure target track signal; using the encoder to perform feature extraction on the mixed audio signal to obtain time-frequency features of the mixed audio signal; based on the time-frequency features, determining scale information of at least one of the attention units; using at least one of the attention units in the separator, separating the time-frequency features based on the scale information of each of the attention units to obtain target track information; using the decoder to decode the target track information to obtain an estimated target track signal; training the track separation model according to the pure target track signal and the estimated target track signal to obtain a trained track separation model.

[0005] As an implementation mode, determining the scale information of at least one of the attention units based on the time-frequency features may include: determining the scale information of the at least one attention unit based on the number of speech frames and the number of frequency bands in the time-frequency features, wherein the scale information of each of the attention units is different, and the scale information includes the number of sub-speech frames and the number of sub-frequency bands, the number of sub-speech frames is less than or equal to the number of speech frames and the number of sub-frequency bands is less than or equal to the number of frequency bands.

[0006] As an implementation mode, the time-frequency features are separated and processed based on the scale information of each attention unit through at least one of the attention units in the separator to obtain the target audio track information, which may include: dividing the target time-frequency range into sub-time-frequency ranges according to the scale information corresponding to each attention unit, wherein the target time-frequency range is formed based on the number of speech frames and the number of frequency bands in the time-frequency features; for any of the attention units, extracting the time-frequency correlation information corresponding to the target audio track information from the target time-frequency features through the attention unit, wherein the target time-frequency features are the features corresponding to the sub-time-frequency range corresponding to the attention unit in the time-frequency features; and generating the target audio track information based on the time-frequency correlation information obtained by each of the attention units.

[0007] As an implementation method, generating the target audio track information based on the time-frequency correlation information obtained by each of the attention units may include: fusing the time-frequency correlation information respectively obtained by each of the attention units to obtain the time-frequency correlation information corresponding to the target audio track information; and generating the target audio track information based on the time-frequency correlation information.

[0008] As an implementation mode, the separator includes a plurality of separators, each of which includes a plurality of attention units, wherein the time-frequency features are separated and processed based on the scale information of each attention unit through at least one of the attention units in the separator to obtain the target audio track information, which may include: for each separator, fusing the time-frequency correlation information respectively obtained by the plurality of attention units in the separator to obtain the time-frequency correlation information for the separator; fusing the time-frequency correlation information for each of the separators to obtain the target time-frequency correlation information; and generating the target audio track information based on the target time-frequency correlation information.

[0009] According to a second aspect of an embodiment of the present disclosure, a method for track separation is provided, which may include: obtaining an audio signal to be processed; inputting the audio signal to be processed into an encoder of a track separation model to obtain time-frequency features of the audio signal to be processed; determining, based on the time-frequency features, scale information of at least one attention unit included in a separator of the track separation model; performing separation processing on the time-frequency features based on the scale information of each attention unit through at least one of the attention units in the separator to obtain target track information; inputting the target track information into a decoder of the track separation model to obtain a target audio signal.

[0010] As an implementation mode, determining the scale information of at least one attention unit included in the separator of the audio track separation model based on the time-frequency features may include: determining the scale information of the at least one attention unit based on the number of speech frames and the number of frequency bands in the time-frequency features, wherein the scale information of each attention unit is different, and the scale information includes the number of sub-speech frames and the number of sub-frequency bands, the number of sub-speech frames is less than or equal to the number of speech frames and the number of sub-frequency bands is less than or equal to the number of frequency bands.

[0011] As an implementation mode, the time-frequency features are separated and processed based on the scale information of each attention unit through at least one of the attention units in the separator to obtain the target audio track information, which may include: dividing the target time-frequency range into sub-time-frequency ranges according to the scale information corresponding to each attention unit, wherein the target time-frequency range is formed based on the number of speech frames and the number of frequency bands in the time-frequency features; for any of the attention units, extracting the time-frequency correlation information corresponding to the target audio track information from the target time-frequency features through the attention unit, wherein the target time-frequency features are the features corresponding to the sub-time-frequency range corresponding to the attention unit in the time-frequency features; and generating the target audio track information based on the time-frequency correlation information obtained by each of the attention units.

[0012] As an implementation method, generating the target audio track information based on the time-frequency correlation information obtained by each of the attention units may include: fusing the time-frequency correlation information respectively obtained by each of the attention units to obtain the time-frequency correlation information corresponding to the target audio track information; and generating the target audio track information based on the time-frequency correlation information.

[0013] As an implementation mode, the separator includes a plurality of separators, each of which includes a plurality of attention units, wherein the time-frequency features are separated and processed based on the scale information of each attention unit through at least one of the attention units in the separator to obtain the target audio track information, which may include: for each separator, fusing the time-frequency correlation information respectively obtained by the plurality of attention units in the separator to obtain the time-frequency correlation information for the separator; fusing the time-frequency correlation information for each of the separators to obtain the target time-frequency correlation information; and generating the target audio track information based on the target time-frequency correlation information.

[0014] According to a third aspect of an embodiment of the present disclosure, a training device for a track separation model is provided, the track separation model comprising an encoder, a separator and a decoder, the separator comprising at least one attention unit, the device may comprise: an acquisition module, configured to acquire a mixed audio signal and a pure target track signal, wherein the mixed audio signal comprises the pure target track signal; a training module, configured to: use the encoder to perform feature extraction on the mixed audio signal to obtain time-frequency features of the mixed audio signal; based on the time-frequency features, determine the scale information of at least one of the attention units; through at least one of the attention units in the separator, perform separation processing on the time-frequency features based on the scale information of each of the attention units to obtain target track information; use the decoder to decode the target track information to obtain an estimated target track signal; train the track separation model according to the pure target track signal and the estimated target track signal to obtain a trained track separation model.

[0015] As an embodiment, the training module can be configured to: determine the scale information of the at least one attention unit based on the number of speech frames and the number of frequency bands in the time-frequency features, wherein the scale information of each attention unit is different, and the scale information includes the number of sub-speech frames and the number of sub-frequency bands, the number of sub-speech frames is less than or equal to the number of speech frames and the number of sub-frequency bands is less than or equal to the number of frequency bands.

[0016] As an implementation method, the training module can be configured as follows: according to the scale information corresponding to each of the attention units, the target time-frequency range is divided into sub-time-frequency ranges, wherein the target time-frequency range is formed based on the number of speech frames and the number of frequency bands in the time-frequency features; for any of the attention units, the time-frequency correlation information corresponding to the target audio track information is extracted from the target time-frequency features through the attention unit, wherein the target time-frequency features are the features corresponding to the sub-time-frequency range corresponding to the attention unit in the time-frequency features; and the target audio track information is generated based on the time-frequency correlation information obtained by each of the attention units.

[0017] As an implementation method, the training module may be configured to: fuse the time-frequency correlation information respectively obtained by each of the attention units to obtain the time-frequency correlation information corresponding to the target audio track information; and generate the target audio track information based on the time-frequency correlation information.

[0018] As an embodiment, the separator includes multiple separators, each separator includes multiple attention units, wherein the training module can be configured to: for each separator, fuse the time-frequency correlation information respectively obtained by the multiple attention units in the separator to obtain the time-frequency correlation information for the separator; fuse the time-frequency correlation information for each of the separators to obtain target time-frequency correlation information; and generate the target audio track information based on the target time-frequency correlation information.

[0019] According to a fourth aspect of an embodiment of the present disclosure, a track separation device is provided, and the device may include: an input module, configured to obtain an audio signal to be processed; an encoding module, configured to input the audio signal to be processed into an encoder of a track separation model to obtain time-frequency features of the audio signal to be processed; a separation module, configured to determine, based on the time-frequency features, scale information of at least one attention unit included in a separator of the track separation model; through at least one of the attention units in the separator, the time-frequency features are separated and processed based on the scale information of each attention unit to obtain target track information; a decoding module, configured to input the target track information into a decoder of the track separation model to obtain the target audio signal.

[0020] As an implementation, the separation module can be configured to: determine the scale information of the at least one attention unit based on the number of speech frames and the number of frequency bands in the time-frequency features, wherein the scale information of each attention unit is different, and the scale information includes the number of sub-speech frames and the number of sub-frequency bands, the number of sub-speech frames is less than or equal to the number of speech frames and the number of sub-frequency bands is less than or equal to the number of frequency bands.

[0021] As an implementation mode, the separation module may be configured as follows: according to the scale information corresponding to each of the attention units, the target time-frequency range is divided into sub-time-frequency ranges, wherein the target time-frequency range is formed based on the number of speech frames and the number of frequency bands in the time-frequency features; for any of the attention units, the time-frequency correlation information corresponding to the target audio track information is extracted from the target time-frequency features through the attention unit, wherein the target time-frequency features are features corresponding to the sub-time-frequency range corresponding to the attention unit in the time-frequency features; and the target audio track information is generated based on the time-frequency correlation information obtained by each of the attention units.

[0022] As an implementation method, the separation module may be configured to: fuse the time-frequency correlation information respectively obtained by each of the attention units to obtain the time-frequency correlation information corresponding to the target audio track information; and generate the target audio track information based on the time-frequency correlation information.

[0023] As an embodiment, the separator includes multiple separators, each separator includes multiple attention units, wherein the separation module can be configured as: for each separator, fusing the time-frequency correlation information respectively obtained by the multiple attention units in the separator to obtain the time-frequency correlation information for the separator; fusing the time-frequency correlation information for each of the separators to obtain the target time-frequency correlation information; and generating the target audio track information based on the target time-frequency correlation information.

[0024] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, which may include: at least one processor; at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to execute the audio track separation method or the model training method as described above.

[0025] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided. When the instructions are executed by at least one processor, the at least one processor is prompted to execute the audio track separation method or model training method as described above.

[0026] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, wherein instructions in the computer program product are executed by at least one processor in an electronic device to perform the audio track separation method or the model training method as described above.

[0027] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0028] By determining the scale information adopted by the attention unit based on the time-frequency characteristics of the mixed audio signal, the audio track signals covering different sound sources can be extracted more accurately.

[0029] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0031] Figure 1 is a flowchart of a method for training a track separation model according to an embodiment of the present disclosure;

[0032] Figure 2 is a schematic diagram of a track separation system according to an embodiment of the present disclosure;

[0033] Figure 3 is a schematic diagram of a division scale according to an embodiment of the present disclosure;

[0034] Figure 4 is a schematic diagram of a residual unit according to an embodiment of the present disclosure;

[0035] Figure 5 is a structural schematic diagram of an audio track separation device according to an embodiment of the present disclosure;

[0036] Figure 6 is a schematic diagram of fusion between multiple separators according to an embodiment of the present disclosure.

[0037] Figure 7 is a flow chart of a method for separating audio tracks according to an embodiment of the present disclosure;

[0038] Figure 8 is a block diagram of a training device for a track separation model according to an embodiment of the present disclosure;

[0039] Fig. 9 is a block diagram of a track separation apparatus according to an embodiment of the present disclosure;

[0040] Fig.10 is a block diagram of an electronic device according to an embodiment of the present disclosure.

[0041] Throughout the drawings, it should be noted that like reference numbers are used to depict the same or similar elements, features, and structures. DETAILED DESCRIPTION

[0042] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0043] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of the embodiments of the present disclosure as defined by the claims and their equivalents. Various specific details are included to assist in understanding, but these details are considered to be exemplary only. Therefore, one of ordinary skill in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, descriptions of well-known functions and structures are omitted for clarity and brevity.

[0044] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it should be clear to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and is not intended to limit the purpose of the present disclosure as defined by the claims and their equivalents.

[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0046] With the rapid development of deep learning technology, the track separation technology based on deep learning has been continuously improved in recent years, and it is possible to generate separated tracks with less distortion. For example, a time domain U-net scheme can be used for track separation, that is, the input time domain signal passes through an encoder composed of a multi-layer convolutional network, and then uses a two-layer bidirectional long short-term memory (LSTM) network to separate the time domain signal, and finally generates the final separation signal through a decoder composed of a multi-layer convolutional network. However, the separation module in this scheme adopts a bidirectional long short-term memory network LSTM structure. Although it can model the temporal correlation in the signal to a certain extent, it cannot model the correlation between different frequencies in the signal well. In addition, a frequency domain separation scheme using an encoder, a separation module, and a decoder can also be used. Among them, the encoder and the decoder adopt a convolutional neural network structure, and the separation module adopts both the time domain attention mechanism and the frequency domain attention mechanism, which can better extract the time-frequency correlation information. However, this solution is used in speech enhancement systems, and the overall system design cannot handle high sampling rate signals well. In addition, the time-frequency attention mechanism of this solution only uses two fixed scales: full frequency domain and full time domain, and cannot handle music signals covering many different types of instruments well.

[0047] In order to better utilize the time-frequency correlation in the separated signals during the process of track separation, this paper proposes a track separation scheme based on a multi-scale attention mechanism. After the input audio is extracted from the time-frequency features by the encoder, the attention mechanism is used to model the time-frequency correlation between signals based on time-frequency units of different scales. Finally, the audio information extracted by the attention mechanisms of different scales is fused together, and the separate tracks are extracted by the decoder, thereby improving the track separation effect.

[0048] Hereinafter, according to various embodiments of the present disclosure, the method, device, and system of the present disclosure will be described in detail with reference to the accompanying drawings.

[0049] Figure 11 is a flow chart of a method for training a track separation model according to an embodiment of the present disclosure. The track separation model according to an embodiment of the present disclosure can be used to perform track separation on any audio signal, and can obtain a track signal with high fidelity.

[0050] The training method of the track separation model according to the present disclosure can be performed by any electronic device. The electronic device can be at least one of a smart phone, a tablet computer, a portable computer, and a desktop computer. The electronic device can be installed with a target application for implementing the training method of the present disclosure.

[0051] The audio track separation model disclosed in the present invention may include an encoder, a separator and a decoder. The encoder, the separator and the decoder may be implemented based on a deep learning approach. The separator may adopt a multi-scale attention mechanism when extracting the time-frequency correlation information of the input signal, which can not only better process audio signals covering different types of sound sources but also obtain separated audio tracks with high fidelity.

[0052] Reference Figure 1 , in step S101, a mixed audio signal and a pure target track signal are obtained. The mixed audio signal includes the pure target track signal. Different training samples can be used according to model requirements. For example, if a track separation model capable of outputting a specific track signal is desired, the pure target track signal is a training sample for the specific track signal. If a track separation model capable of outputting multiple specific track signals is desired, the pure target track signal is a training sample for these specific track signals. The mixed audio signal can be obtained by mixing the pure target track signal with other audio signals. For example, the pure target track signal is a pure bass signal, and the mixed audio signal is a song including the bass signal. The mixed audio signal can be used as input of the track separation model for data processing, and the parameters of the track separation model can be adjusted using the output of the pure target track signal and the track separation model.

[0053] In step S102, an encoder of the track separation model is used to extract features of the mixed audio signal to obtain time-frequency features of the mixed audio signal.

[0054] The input and output of the audio track separation model can directly use time domain signals, or the input can use frequency domain signals that have undergone short-time Fourier transform, and then the output can be converted from frequency domain signals to time domain signals through short-time inverse Fourier transform. The following description takes the model input as a frequency domain signal as an example.

[0055] Before inputting the audio track separation model, the mixed audio signal to be processed can be subjected to short-time Fourier transform to obtain the spectrum signal of the mixed audio signal. The spectrum signal passes through the encoder to obtain the time-frequency characteristics of the mixed audio signal.

[0056] The encoder may adopt a structure based on a multi-layer two-dimensional convolutional network. For example, the encoder may be composed of multiple superimposed encoding units, each of which may include a residual unit and a two-dimensional convolutional unit. The two-dimensional convolutional unit may include a two-dimensional convolutional layer, a regularization operation layer (such as a batch normalization operation), and an activation function layer (such as a ReLu function). The residual unit may be composed of two two-dimensional convolutional units plus a residual connection, such as Figure 4 shown.

[0057] As an example, the spectrum signal input to the encoder is X T×F×N , where X represents the spectrum signal, T represents the number of speech frames, F represents the number of frequency bands of the spectrum, and N represents the number of input channels. For example, for a single-channel complex spectrum, including the real and imaginary parts of the spectrum, N is equal to 2. The spectrum signal X T×F×N After passing through the encoder, the output of the encoder (that is, the time-frequency characteristics of the mixed audio signal) is Y T′×F′×C , where Y represents the time-frequency feature, T′ and F′ represent the number of speech frames and frequency bands output by the encoder (i.e., the number of time domain points and frequency domain points after encoding), respectively, and C represents the number of channels output by the encoder.

[0058] In step S103, based on the time-frequency characteristics of the mixed audio signal, scale information of at least one attention unit in the separator of the track separation model is determined.

[0059] In an embodiment of the present application, the separator may include at least one attention unit, wherein the attention unit is a unit that uses an attention mechanism to focus attention on time-frequency features within a specific range to perform separation processing of the time-frequency features. The scale information of at least one attention unit may be determined based on the number of speech frames and the number of frequency bands in the time-frequency features of the mixed audio signal, and the scale information of each attention unit is different.

[0060] The scale information may be determined based on the number of speech frames and the number of frequency bands in the time-frequency characteristics of the mixed audio signal, and the number of sub-speech frames in the scale information is less than or equal to the number of speech frames and the number of sub-frequency bands in the scale information is less than or equal to the number of frequency bands.

[0061] Assume that the time-frequency range of the separator input is T′×F′, T′ and F′ respectively represent the number of speech frames and the number of frequency bands output by the encoder, and T″×F″ is the scale information of the attention unit dividing the time-frequency range. When determining the division scale, the scale information used by the attention unit can be determined based on the following conditions: 1≤T″≤T′ and 1≤F″≤F′. Here, the scale can be understood as the unit size used to divide the input time-frequency range. In the present disclosure, scale and scale information have the same meaning.

[0062] In the case where the separator includes a plurality of attention units, each attention unit may use different scale information to divide the time-frequency range corresponding to the time-frequency feature of the mixed audio signal. In this way, the attention unit may extract the corresponding time-frequency feature in the divided time-frequency range.

[0063] Figure 3 Schematic diagram of the division scale according to an embodiment of the present disclosure. Figure 3 , 301 represents the time-frequency range T′×F′ of the separator input, and each small dashed box in 301 represents a different division scale, such as scale 302, 303, 304, 305. A size can be assigned to each attention unit so that the attention unit is divided using the scale. Figure 3 The multi-scale partitioning shown is applicable to all different attention mechanisms.

[0064] In step S104, at least one attention unit in the separator is used to separate the time-frequency features of the mixed audio signal based on the scale information of each attention unit to obtain target audio track information.

[0065] The separator can adopt a multi-scale attention mechanism to extract the time-frequency correlation information of the mixed audio signal and generate the target audio track information based on the time-frequency correlation information.

[0066] According to the scale information determined above, the target time-frequency range can be divided into sub-time-frequency ranges, where the target time-frequency range can be formed based on the number of speech frames and the number of frequency bands in the time-frequency features of the mixed audio signal. Then, the time-frequency correlation information corresponding to the target track information is extracted from the corresponding features of the sub-time-frequency range in the time-frequency features of the mixed audio signal, and the target track information is generated based on the time-frequency correlation information. For example, the time-frequency correlation information extracted by applying the attention mechanism on different scale ranges can be fused together to obtain the target track information. Here, the target track information can be used to generate the corresponding masking matrix in the decoder later.

[0067] As an example, a separator may include multiple attention units, and the scale information of the attention unit applying the attention mechanism may be determined based on the time-frequency characteristics of the mixed audio signal, and the attention unit may divide the time-frequency range corresponding to the time-frequency characteristics according to the determined scale information, and use the attention mechanism to obtain the time-frequency correlation information of the mixed audio signal in the divided sub-ranges. During the model training process, the number of attention units included in the separator and the scale information used by each attention unit may be pre-set. The number of attention units may be set differently according to design requirements.

[0068] The time-frequency correlation information obtained by multiple attention units can be fused, and the fused time-frequency correlation information can be used as the time-frequency correlation information of the mixed audio signal, and then the target audio track information is generated based on the time-frequency correlation information of the mixed audio signal.

[0069] According to another example of the present disclosure, the track separation model may include multiple separators, each of which may include multiple attention units. For each separator, the time-frequency correlation information obtained by the multiple attention units in the separator may be fused to obtain the time-frequency correlation information of the mixed audio signal, and then the target track information is generated based on the fusion of the output information of each separator. Each attention unit in the separator may use different scale information to divide the time-frequency range corresponding to the time-frequency features of the mixed audio signal, and the scale information used by the attention unit in one separator may be at least partially different from the scale information used by the attention unit in another separator.

[0070] As another example, multiple separators can be used as a neural network layer in the track separation model. For each layer, the output results of multiple separators in the layer can be fused, and the fusion results can be applied to multiple separators in the next layer. In different separators, multiple attention units can be used, and each attention unit can use different scales.

[0071] By adopting a multi-scale attention mechanism, the time-frequency correlation in the mixed audio signal can be extracted more effectively, making the prediction of the trained track separation model more accurate.

[0072] In step S105, the decoder of the track separation model is used to decode the target track information to obtain an estimated target track signal. The decoder may be composed of a plurality of decoding units, each of which may include a two-dimensional deconvolution layer and a residual unit. The above examples are only exemplary, and the present disclosure is not limited thereto.

[0073] As an example, based on the target audio track information, a target masking matrix of the target audio track signal can be obtained through a decoder. After the spectrum signal of the mixed audio signal is multiplied by the target masking matrix, an estimated target audio track signal, such as vocals, drums, bass and other musical instruments, can be obtained through a short-time inverse Fourier transform.

[0074] As another example, a target audio track spectrum may be obtained through a decoder based on the target audio track information, and an estimated target audio track signal may be obtained through short-time inverse Fourier transform.

[0075] In step S106, a track separation model is trained according to the pure target track signal and the estimated target track signal to obtain a trained track separation model.

[0076] The time domain features or frequency domain features of the pure target track signal and the estimated target track signal can be used to construct a loss function, and the parameters of the track separation model can be updated by minimizing the loss calculated by the loss function. In the case of using frequency domain features, the constructed loss function can be, for example, a spectrum mean square error function, a logarithmic energy spectrum mean absolute error function, and in the case of using time domain features, a time domain average error function can be constructed as the loss function of the training model. With the goal of minimizing the loss obtained by the loss function, the parameters of the track separation model are updated until the model converges.

[0077] The present invention adopts a multi-scale attention mechanism to effectively extract the time-frequency correlation in the mixed audio signal, thereby improving the fidelity of the separated target audio track signal and enhancing the overall effect of the model.

[0078] Figure 2 Schematic diagram of a track separation system according to an embodiment of the present disclosure. The track separation system according to an embodiment of the present disclosure may include a short-time Fourier transform (STFT) module, an encoder, a separation module, a decoder and an inverse short-time Fourier transform (ISTFT) module. The following is an example of separating tracks of music signals.

[0079] The input music signal passes through the STFT module to obtain the corresponding music spectrum signal. The music spectrum signal passes through the encoder to obtain the time-frequency characteristics of the music signal. The time-frequency characteristics output by the encoder pass through the separation module to obtain the target track information of the music signal. The target track information passes through the decoder to obtain the target masking matrix corresponding to the target track signal. After the music spectrum signal is multiplied by the target masking matrix, it passes through the ISTFT module to obtain the target track signal, such as vocals, drums, bass and other musical instruments.

[0080] The encoder can adopt a structure based on a multi-layer two-dimensional convolutional network. Figure 2 As shown in , the encoder can be composed of multiple encoding units stacked together, each encoding unit can include a residual unit and a two-dimensional convolution unit. The two-dimensional convolution unit can include a two-dimensional convolution layer, a regularization operation layer (such as a batch normalization operation) and an activation function layer (such as a ReLu function). The residual unit can be composed of two two-dimensional convolution units plus a residual connection, such as Figure 4 shown.

[0081] As an example, the music spectrum signal input to the encoder is X T×F×N, where T represents the number of speech frames, F represents the number of frequency bands of the spectrum, and N represents the number of input channels. For example, for a single-channel complex spectrum, including the real and imaginary parts of the spectrum, N is equal to 2. Music spectrum signal X T×F×N After passing through the encoder, the output of the encoder is Y T′×F′×C , where T′ and F′ represent the number of speech frames and frequency bands output by the encoder (i.e., the number of time domain points and frequency domain points after encoding), respectively, and C represents the number of channels output by the encoder.

[0082] The separation module can be composed of multiple separators. For each separator, the input can first pass through two residual units to extract features, then pass through multiple attention units to extract time-frequency correlation information, and then concatenate the time-frequency correlation information extracted by each attention unit and the output of the residual unit (i.e., implemented by Concat), and then pass through a two-dimensional convolution unit to obtain the output result.

[0083] For example, for Figure 2 In the first separator, the time-frequency features output by the encoder are extracted through two residual units, and the time-frequency correlation information is extracted through multiple attention units. Then, the time-frequency correlation information extracted by each attention unit and the output of the residual unit are concatenated, and then the output result of the first separator is obtained through a two-dimensional convolution unit. Assuming that the separation module includes a separator, the output result of the separator is the target track information. For the subsequent separators, the output result of the previous separator can be used as its input to continue the operation.

[0084] also, Figure 2 It is shown that the separation module includes a plurality of separators and each separator includes a plurality of attention units, but the present disclosure is not limited thereto, for example, the separation module may include a plurality of separators and each separator includes one attention unit, or the separation module includes one separator and the separator includes a plurality of attention units. The above examples are merely exemplary, and the present disclosure is not limited thereto.

[0085] According to an embodiment of the present disclosure, a multi-scale self-attention mechanism can be used to extract the time-frequency correlation information of a music signal. Each attention unit can use different scales to divide the time-frequency features of the music signal, and then extract the time-frequency correlation information (also referred to as attention information) based on the divided time-frequency features.

[0086] Figure 3 is a schematic diagram of dividing scales according to an embodiment of the present disclosure.

[0087] Assume that the time-frequency range of the separation module input is T′×F′, T′ and F′ represent the number of speech frames and the number of frequency bands output by the encoder respectively, and T″×F″ is the scale of the self-attention mechanism applied by the attention unit, that is, the scale of the time-frequency range of the input divided by the attention unit. When determining the division scale, the scale information used by the attention unit can be determined based on the following conditions: 1≤T″≤T′ and 1≤F″≤F′. Here, the scale can be understood as the unit scale used to divide the input time-frequency range.

[0088] Reference Figure 3 , 301 represents the time-frequency range T′×F′ of the separation module input, and each small dashed box in 301 represents a different division scale, such as scales 302, 303, 304, and 305. Figure 3 The division scale is only exemplary, and the present disclosure is not limited thereto. After determining the division scale, the attention unit can divide the input time-frequency range according to the determined scale, and then use each divided small time-frequency range to extract the time-frequency correlation information of the music signal.

[0089] In order to make the operation easier when dividing the time-frequency range, T′ and F′ can be made divisible by T″ and F″ respectively.

[0090] Assuming that the value of T′×F′ is 120×256, the possible values ​​of T″ and F″ are as shown in the following table.

[0091] T″ 1 1 120 2 4 8 15 1 30 F″ 1 256 1 128 64 32 16 64 1

[0092] Among them, T″×F″=1×1 can represent the calculation of the correlation of each time-frequency point. T″×F″=T′1 represents the calculation of the frequency domain correlation of the entire signal. T″×F″=1×F′ represents the calculation of the time domain correlation of the entire signal. Represents that only the time domain correlation of one quarter of the frequency band is calculated. The above value range is only exemplary, and the present disclosure is not limited thereto. For example, only the time domain correlation of one quarter of the frequency band is considered, mainly because the signal range of some musical instruments does not cover the entire frequency band. Only the overall correlation can be calculated through the full frequency band, and the correlation of the signal itself cannot be accurately measured. For example, the bass signal generally only appears in the low frequency range, so it is more accurate to calculate the correlation in the frequency band range of the bass.

[0093] Therefore, the present disclosure adopts a multi-scale attention mechanism, which can not only calculate the correlation of the entire music signal, but also take into account the complexity of the signal and use more scales to more accurately calculate the signal correlation of various sound sources.

[0094] After determining the scale of each attention unit, the self-attention mechanism can be used to extract the time-frequency correlation of the music signal.

[0095] Assume that the input data of the attention unit is ZT′×F′×C According to the determined scale T″×F″, the input data can be converted into Z M×T″×F″×C , where M is the total number of divisions (i.e. the number of hourly frequency ranges divided from the input time-frequency range T′×F′ according to the scale T″×F″), which satisfies:

[0096] T′×F′=M×T″×F″

[0097] Assumption C in =T″×F″×C, then the input data can be expressed as According to the self-attention mechanism, the following variables can be obtained through transformation:

[0098] Q=f Q (Z) = Z·W Q

[0099] K=f K (Z) = Z·W K

[0100] V=f V (Z) = Z·W V

[0101] Among them, Q, K, and V represent the query, key, and value in the attention mechanism respectively, and the dimensions of Q, K, and V are M×C att ,C att is the scalar parameter in the attention mechanism, f Q , f K and f V Represents the calculation functions of Q, K and V respectively, which can be obtained by inputting data Z and parameter matrix W Q , W K , W V Multiply to get Q, K, V values, W Q , W K , W V The dimension is C in ×C att , the softmax output S in the attention mechanism is calculated as follows:

[0102]

[0103] Among them, A(Q, K, V) represents the attention calculation function based on Q, K, V. represents the transpose of the matrix K, softmax() represents the softmax function, and sqrt() represents the square root operation. Finally, S is converted into the output Z of the final attention mechanism through the following transformation att , so that the output Z attHas the same dimensions as the input Z: Z att =S×W S , W S It is the parameter matrix after softmax.

[0104] Each attention unit can obtain the time-frequency correlation information of the music signal according to the above-mentioned method of calculating attention.

[0105] When a separator includes multiple attention units, each attention unit can use different scales to divide the time-frequency range, and then the output results of all attention units in the separator are spliced ​​to obtain the time-frequency correlation information of the music signal. Figure 2 As shown, a separator includes multiple attention units, and the target audio track information calculated by the separator can be obtained by splicing the output results of each attention unit.

[0106] Figure 2 The case where each neural network layer includes only one separator is shown, but in order to better extract the time-frequency correlation, each layer may include multiple separators. The output results of multiple separators in each layer are fused, and the fusion results are applied to multiple separators in the next layer. The scales used by the attention units in different separators can also be different.

[0107] Figure 6 is a schematic diagram of the fusion between multiple separators according to an embodiment of the present disclosure. Figure 6 In the figure, each dotted box may represent a neural network layer in the audio track separation model. The separation module may include multiple layers. Each layer of the separation module may include two separators. When the two separators of the next layer calculate the time-frequency correlation, each separator of the next layer may use the fusion result of the two separators of the previous layer to further extract the time-frequency correlation of the music signal. Figure 6 The number of separators included in each layer is merely exemplary, and the present disclosure is not limited thereto.

[0108] After obtaining the target audio track signal, the target masking matrix of the music signal can be obtained through the decoder, and the target audio track signal, that is, the desired audio signal, can be obtained based on the target masking matrix. Figure 2 As shown, the decoder may be composed of a plurality of decoding units stacked together, and each decoding unit may include a two-dimensional deconvolution layer and a residual unit. The above examples are only exemplary, and the present disclosure is not limited thereto.

[0109] Figure 5 It is a structural diagram of the audio track separation device in the hardware operating environment of the embodiment of the present disclosure.

[0110] like Figure 5As shown, the track separation device 500 may include: a processing component 501, a communication bus 502, a network interface 503, an input / output interface 504, a memory 505, and a power supply component 506. Among them, the communication bus 502 is used to realize the connection and communication between these components. The input / output interface 504 may include a video display (such as a liquid crystal display), a microphone and a speaker, and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.), and optionally, the input / output interface 504 may also include a standard wired interface and a wireless interface. The network interface 503 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity interface). The memory 505 may be a high-speed random access memory or a stable non-volatile memory. The memory 505 may also be a storage device independent of the aforementioned processing component 501.

[0111] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the audio track separation device 500, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0112] like Figure 5 As shown, the memory 505 as a storage medium may include an operating system (such as a MAC operating system), a data storage module, a network communication module, a user interface module, a program, and a database.

[0113] exist Figure 5 In the audio track separation device 500 shown, the network interface 503 is mainly used for data communication with external electronic devices / terminals; the input and output interface 504 is mainly used for data interaction with users; the processing component 501 and the memory 505 in the audio track separation device 500 can be set in the audio track separation device 500, and the audio track separation device 500 calls the program stored in the memory 505 and various APIs provided by the operating system through the processing component 501 to execute the audio track separation method or model training method provided in the embodiment of the present disclosure.

[0114] The processing component 501 may include at least one processor, and a computer executable instruction set is stored in the memory 505. When the computer executable instruction set is executed by at least one processor, the audio track separation method or model training method according to the embodiment of the present disclosure is executed. However, the above examples are only exemplary, and the present disclosure is not limited thereto.

[0115] The processing component 501 can control the components included in the audio track separation device 500 by executing a program.

[0116] As an example, the track separation device 500 can be a PC computer, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the track separation device 500 is not necessarily a single electronic device, but can also be any device or circuit that can execute the above-mentioned instructions (or instruction sets) individually or in combination. The track separation device 500 can also be part of an integrated control system or system manager, or can be configured as a portable electronic device that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0117] In the track separation device 500, the processing component 501 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not a limitation, the processing component 501 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0118] The processing component 501 can run instructions or codes stored in the memory, wherein the memory 505 can also store data. Instructions and data can also be sent and received over the network via the network interface 503, wherein the network interface 503 can use any known transmission protocol.

[0119] The memory 505 may be integrated with the processing component 501, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 505 may include a separate device, such as an external disk drive, a storage array, or any other storage device that may be used by a database system. The memory and the processing component 501 may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processing component 501 can read data stored in the memory 505.

[0120] Figure 7 is a flowchart of a method for separating audio tracks according to an embodiment of the present disclosure. Figure 7 The illustrated audio track separation method may be executed by any of the electronic devices described above.

[0121] Reference Figure 7 In step S701, an audio signal to be processed is obtained, for example, a music signal.

[0122] In step S702, the audio signal to be processed is input into an encoder of the audio track separation model to obtain the time-frequency features of the audio signal to be processed.

[0123] In step S703, based on the time-frequency characteristics of the audio signal to be processed, scale information of at least one attention unit included in the separator of the track separation model is determined.

[0124] As an example, based on the number of speech frames and the number of frequency bands in the time-frequency features, the scale information of at least one attention unit is determined respectively, wherein the scale information of each attention unit is different, and the scale information includes the number of sub-speech frames and the number of sub-frequency bands, the number of sub-speech frames is less than or equal to the number of speech frames, and the number of sub-frequency bands is less than or equal to the number of frequency bands.

[0125] In step S704, at least one attention unit in the separator is used to separate the time-frequency features based on the scale information of each attention unit to obtain the target audio track information.

[0126] According to the scale information corresponding to each attention unit, the target time-frequency range is divided into sub-time-frequency ranges, wherein the target time-frequency range is formed based on the number of speech frames and the number of frequency bands in the time-frequency feature. For any attention unit, the time-frequency correlation information corresponding to the target track information is extracted from the target time-frequency feature through the attention unit, wherein the target time-frequency feature is the feature corresponding to the sub-time-frequency range corresponding to the attention unit in the time-frequency feature; and the target track information is generated based on the time-frequency correlation information obtained by each attention unit.

[0127] In the case of including multiple attention units, the time-frequency correlation information respectively obtained by each attention unit can be fused to obtain the time-frequency correlation information corresponding to the target track information, and the target track information is generated based on the time-frequency correlation information.

[0128] As another example, in the case where the audio track separation model includes multiple separators and each separator includes multiple attention units, the time-frequency correlation information respectively obtained by the multiple attention units in the separator can be fused for each separator to obtain the time-frequency correlation information for the separator, and the time-frequency correlation information for each separator can be fused to obtain the target time-frequency correlation information; and the target audio track information is generated based on the target time-frequency correlation information.

[0129] In step S705, the target audio track information is input into the decoder of the audio track separation model to obtain the target audio signal.

[0130] For example, the time-frequency correlation information extracted by the attention mechanism is Figure 2 The output of the separation module can be understood as a feature containing correlation information extracted by the neural network. After the feature is input into the decoder and combined with the intermediate information of the generator, the masking matrix corresponding to the target audio signal can be obtained.

[0131] According to another embodiment of the present disclosure, an audio signal to be processed is obtained. For example, a music signal is obtained. The audio signal to be processed can then be input into a track separation model to obtain a target track signal. Here, the track separation model can be obtained using the model training method described above.

[0132] By using the audio track separation method disclosed in the present invention, audio track signals of various sound sources with less distortion can be obtained.

[0133] Figure 8 is a block diagram of a training device for a track separation model according to an embodiment of the present disclosure. Figure 8 , the training device 800 may include an acquisition module 801 and a training module 802. Each module in the training device 800 may be implemented by one or more modules, and the name of the corresponding module may vary according to the type of the module. In various embodiments, some modules in the training device 800 may be omitted, or additional modules may be further included. In addition, the modules / elements according to various embodiments of the present disclosure may be combined to form a single entity, and thus may equivalently perform the functions of the corresponding modules / elements before the combination.

[0134] The track separation model may include an encoder, a separator, and a decoder, and the separator may include at least one attention unit.

[0135] The acquisition module 801 can acquire a mixed audio signal and a clean target audio track signal, wherein the mixed audio signal includes the clean target audio track signal.

[0136] The training module 802 can use an encoder to extract features of the mixed audio signal to obtain the time-frequency features of the mixed audio signal; based on the time-frequency features, determine the scale information of at least one attention unit; through at least one attention unit in the separator, separate the time-frequency features based on the scale information of each attention unit to obtain target audio track information; use a decoder to decode the target audio track information to obtain an estimated target audio track signal; train the audio track separation model according to the pure target audio track signal and the estimated target audio track signal to obtain a trained audio track separation model.

[0137] The scale information of the attention unit applying the attention mechanism can be determined based on the time-frequency characteristics of the mixed audio signal, and the time-frequency range corresponding to the time-frequency characteristics can be divided according to the scale information and the attention mechanism can be used to obtain the time-frequency correlation information of the mixed audio signal, and the time-frequency correlation information can be used to generate the target audio track information.

[0138] The training module 802 can determine the scale information of at least one attention unit based on the number of speech frames and the number of frequency bands in the time-frequency features, wherein the scale information of each attention unit is different, and the scale information may include the number of sub-speech frames and the number of sub-frequency bands, the number of sub-speech frames is less than or equal to the number of speech frames, and the number of sub-frequency bands is less than or equal to the number of frequency bands.

[0139] As an example, the scale information may be determined based on the number of speech frames and the number of frequency bands in the time-frequency features of the mixed audio signal, and the number of sub-speech frames included in the scale information is less than or equal to the number of speech frames of the mixed audio signal and the number of sub-frequency bands in the scale information is less than or equal to the number of frequency bands of the mixed audio signal. The training module 802 may determine the scale information of at least one attention unit based on the number of speech frames and the number of frequency bands in the time-frequency features of the mixed audio signal, respectively, so that the scale information of each attention unit is different.

[0140] The training module 802 can divide the target time-frequency range into sub-time-frequency ranges according to the scale information corresponding to each attention unit, wherein the target time-frequency range is formed based on the number of speech frames and the number of frequency bands in the time-frequency feature; for any attention unit, the time-frequency correlation information corresponding to the target audio track information is extracted from the target time-frequency feature through the attention unit, wherein the target time-frequency feature is the feature corresponding to the sub-time-frequency range corresponding to the attention unit in the time-frequency feature; and the target audio track information is generated based on the time-frequency correlation information obtained by each attention unit.

[0141] The training module 802 can divide the target time-frequency range according to the determined scale information to obtain sub-time-frequency ranges, wherein the target time-frequency range is formed based on the number of speech frames and the number of frequency bands in the time-frequency features; extract the time-frequency correlation information corresponding to the target audio track information from the corresponding features of the sub-time-frequency range in the time-frequency features; and generate the target audio track information based on the time-frequency correlation information.

[0142] The training module 802 may fuse the time-frequency correlation information obtained by each attention unit to obtain the time-frequency correlation information corresponding to the target audio track information, and then generate the target audio track information based on the time-frequency correlation information.

[0143] According to an embodiment of the present disclosure, a separator may include multiple attention units, and each attention unit may use different scale information to divide the time-frequency range corresponding to the time-frequency features of the mixed audio signal. In this case, the training module 802 may fuse the time-frequency correlation information obtained by the multiple attention units, and use the fused time-frequency correlation information as the time-frequency correlation information of the mixed audio signal, and generate the target audio track information based on the time-frequency correlation information of the mixed audio signal.

[0144] The separator of the present disclosure may include multiple separators, each of which includes multiple attention units. In this case, for each separator, the training module 802 may fuse the time-frequency correlation information respectively obtained by the multiple attention units in the separator to obtain the time-frequency correlation information for the separator, fuse the time-frequency correlation information for each separator to obtain the target time-frequency correlation information, and then generate the target track information based on the target time-frequency correlation information.

[0145] According to another embodiment of the present disclosure, a plurality of separators may be included, each separator including a plurality of attention units. In this case, the training module 802 may fuse the time-frequency correlation information obtained by the plurality of attention units in each separator to obtain the time-frequency correlation information of the mixed audio signal, and generate the target track information based on the fusion of the output information of each separator. Each attention unit in the separator may use different scale information to divide the time-frequency range corresponding to the time-frequency features of the mixed audio signal, and the scale information used by the attention unit in one separator may be at least partially different from the scale information used by the attention unit in another separator.

[0146] The above is based on Figures 1 to 4 and Figure 6 The training process of the audio track separation model is described in detail and will not be described here.

[0147] Fig. 9 is a block diagram of a track separation device according to an embodiment of the present disclosure. Fig. 9 , the track separation device 900 may include an input module 901, an encoding module 902, a separation module 903, and a decoding module 904. Each module in the track separation device 900 may be implemented by one or more modules, and the name of the corresponding module may vary according to the type of the module. In various embodiments, some modules in the track separation device 900 may be omitted, or additional modules may be further included. In addition, the modules / elements according to various embodiments of the present disclosure may be combined to form a single entity, and thus may equivalently perform the functions of the corresponding modules / elements before the combination.

[0148] The input module 901 can obtain an audio signal to be processed, such as a music signal.

[0149] The encoding module 902 may input the audio signal to be processed into an encoder of the audio track separation model to obtain the time-frequency characteristics of the audio signal to be processed.

[0150] The separation module 903 can determine the scale information of at least one attention unit included in the separator of the audio track separation model based on the time-frequency features; through at least one attention unit in the separator, the time-frequency features are separated based on the scale information of each attention unit to obtain the target audio track information.

[0151] The decoding module 904 may input the target audio track information into a decoder of the audio track separation model to obtain a target audio signal.

[0152] As an implementation method, the separation module 903 can determine the scale information of at least one attention unit based on the number of speech frames and the number of frequency bands in the time-frequency features, wherein the scale information of each attention unit is different, and the scale information includes the number of sub-speech frames and the number of sub-frequency bands, the number of sub-speech frames is less than or equal to the number of speech frames, and the number of sub-frequency bands is less than or equal to the number of frequency bands.

[0153] As an implementation method, the separation module 903 can divide the target time-frequency range into sub-time-frequency ranges according to the scale information corresponding to each attention unit, wherein the target time-frequency range is formed based on the number of speech frames and the number of frequency bands in the time-frequency feature; for any attention unit, the time-frequency correlation information corresponding to the target audio track information is extracted from the target time-frequency feature through the attention unit, wherein the target time-frequency feature is the feature corresponding to the sub-time-frequency range corresponding to the attention unit in the time-frequency feature; and the target audio track information is generated based on the time-frequency correlation information obtained by each attention unit.

[0154] As an implementation method, the separation module 903 may fuse the time-frequency correlation information respectively obtained by each attention unit to obtain the time-frequency correlation information corresponding to the target audio track information; and generate the target audio track information based on the time-frequency correlation information.

[0155] As an implementation, the track separation model includes multiple separators, each of which includes multiple attention units. In this case, the separation module 903 can fuse the time-frequency correlation information obtained by the multiple attention units in each separator to obtain the time-frequency correlation information for the separator; fuse the time-frequency correlation information for each separator to obtain the target time-frequency correlation information; and generate the target track information based on the target time-frequency correlation information.

[0156] According to another embodiment of the present disclosure, the audio separation device may include an input module and a processing module. After the input module receives the audio signal to be processed, the processing module may input the audio signal to be processed into a track separation model to obtain a target track signal. The track separation model may be obtained using the training method described above.

[0157] According to an embodiment of the present disclosure, an electronic device may be provided. Fig.10 It is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device 1000 may include at least one memory 1002 and at least one processor 1001. The at least one memory 1002 stores a set of computer-executable instructions. When the computer-executable instruction set is executed by the at least one processor 1001, the audio track separation method or model training method according to the embodiment of the present disclosure is executed.

[0158] Processor 1001 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, processor 1001 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0159] The memory 1002 as a storage medium may include an operating system, a data storage module, a network communication module, a user interface module, a program for executing the audio track separation method or model training method of the present disclosure, and a database.

[0160] The memory 1002 may be integrated with the processor 1001, for example, a RAM or flash memory may be arranged within an integrated circuit microprocessor or the like. In addition, the memory 1002 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 1002 and the processor 1001 may be operationally coupled, or may communicate with each other, for example, via an I / O port, a network connection, etc., so that the processor 1001 can read files stored in the memory 1002.

[0161] In addition, the electronic device 1000 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 1000 may be connected to each other via a bus and / or a network.

[0162] Those skilled in the art will understand that Fig.10 The structures shown in the drawings do not constitute limitations of the drawings and may include more or less components than shown in the drawings, or combine certain components, or arrange the components differently.

[0163] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the audio track separation method or model training method according to the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0164] According to an embodiment of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by a processor of a computer device to complete the above-mentioned audio track separation method or model training method.

[0165] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0166] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for separating audio tracks, characterized in that: The method comprises: Obtaining the audio signal to be processed; Inputting the audio signal to be processed into an encoder of a track separation model to obtain time-frequency features of the audio signal to be processed; Determining, based on the time-frequency features, scale information of at least one attention unit included in the separator of the track separation model; By using at least one of the attention units in the separator, the time-frequency features are separated and processed based on the scale information of each of the attention units to obtain target track information; Input the target audio track information into the decoder of the audio track separation model to obtain the target audio signal, The scale information includes the number of sub-speech frames and the number of sub-frequency bands, the number of sub-speech frames is less than or equal to the number of speech frames in the time-frequency feature, and the number of sub-frequency bands is less than or equal to the number of frequency bands in the time-frequency feature.

2. The method according to claim 1, characterized in that Determining, based on the time-frequency features, scale information of at least one attention unit included in the separator of the track separation model, comprising: Based on the number of speech frames and the number of frequency bands in the time-frequency features, the scale information of the at least one attention unit is determined respectively, wherein the scale information of each attention unit is different.

3. The method according to claim 1, characterized in that The method further comprises: performing separation processing on the time-frequency features based on the scale information of each attention unit by at least one attention unit in the separator to obtain target track information, including: According to the scale information corresponding to each of the attention units, the target time-frequency range is divided into sub-time-frequency ranges, wherein the target time-frequency range is formed based on the number of speech frames and the number of frequency bands in the time-frequency features; For any of the attention units, extracting time-frequency correlation information corresponding to the target audio track information from the target time-frequency feature through the attention unit, wherein the target time-frequency feature is a feature corresponding to the sub-time-frequency range corresponding to the attention unit in the time-frequency feature; and The target audio track information is generated based on the time-frequency correlation information obtained by each of the attention units.

4. The method according to claim 3, characterized in that Generating the target audio track information based on the time-frequency correlation information obtained by each of the attention units includes: The time-frequency correlation information respectively obtained by each of the attention units is integrated to obtain the time-frequency correlation information corresponding to the target audio track information; The target audio track information is generated based on the time-frequency correlation information.

5. The method according to any one of claims 1 to 3, characterized in that The separator includes a plurality of separators, each separator includes a plurality of attention units, The step of separating the time-frequency features based on the scale information of each attention unit through at least one attention unit in the separator to obtain target track information includes: For each separator, the time-frequency correlation information respectively obtained by multiple attention units in the separator is fused to obtain the time-frequency correlation information for the separator; The time-frequency correlation information of each separator is merged to obtain target time-frequency correlation information; The target audio track information is generated based on the target time-frequency correlation information.

6. A method for training a track separation model, characterized in that: The track separation model includes an encoder, a separator and a decoder, the separator includes at least one attention unit, and the method includes: Acquire a mixed audio signal and a clean target audio track signal, wherein the mixed audio signal includes the clean target audio track signal; Using the encoder to extract features from the mixed audio signal to obtain time-frequency features of the mixed audio signal; Based on the time-frequency features, determining scale information of at least one of the attention units; By using at least one of the attention units in the separator, the time-frequency features are separated and processed based on the scale information of each of the attention units to obtain target track information; Decoding the target audio track information using the decoder to obtain an estimated target audio track signal; The track separation model is trained according to the pure target track signal and the estimated target track signal to obtain a trained track separation model, The scale information includes the number of sub-speech frames and the number of sub-frequency bands, the number of sub-speech frames is less than or equal to the number of speech frames in the time-frequency feature, and the number of sub-frequency bands is less than or equal to the number of frequency bands in the time-frequency feature.

7. The method according to claim 6, characterized in that Determining scale information of at least one of the attention units based on the time-frequency features includes: Based on the number of speech frames and the number of frequency bands in the time-frequency features, the scale information of the at least one attention unit is determined respectively, wherein the scale information of each attention unit is different.

8. The method according to claim 6, characterized in that The method further comprises: performing separation processing on the time-frequency features based on the scale information of each attention unit by at least one attention unit in the separator to obtain target track information, including: According to the scale information corresponding to each of the attention units, the target time-frequency range is divided into sub-time-frequency ranges, wherein the target time-frequency range is formed based on the number of speech frames and the number of frequency bands in the time-frequency features; For any of the attention units, extracting time-frequency correlation information corresponding to the target audio track information from the target time-frequency feature through the attention unit, wherein the target time-frequency feature is a feature corresponding to the sub-time-frequency range corresponding to the attention unit in the time-frequency feature; and The target audio track information is generated based on the time-frequency correlation information obtained by each of the attention units.

9. The method according to claim 8, characterized in that Generating the target audio track information based on the time-frequency correlation information obtained by each of the attention units includes: The time-frequency correlation information respectively obtained by each of the attention units is integrated to obtain the time-frequency correlation information corresponding to the target audio track information; The target audio track information is generated based on the time-frequency correlation information.

10. The method according to any one of claims 6 to 8, characterized in that The separator includes a plurality of separators, each separator includes a plurality of attention units, The step of separating the time-frequency features based on the scale information of each attention unit through at least one attention unit in the separator to obtain target track information includes: For each separator, the time-frequency correlation information respectively obtained by multiple attention units in the separator is fused to obtain the time-frequency correlation information for the separator; The time-frequency correlation information of each separator is merged to obtain target time-frequency correlation information; The target audio track information is generated based on the target time-frequency correlation information.

11. A track separation device, characterized in that: The device comprises: An input module, configured to obtain an audio signal to be processed; An encoding module, configured to input the audio signal to be processed into an encoder of a track separation model to obtain time-frequency features of the audio signal to be processed; A separation module is configured to determine, based on the time-frequency features, scale information of at least one attention unit included in the separator of the track separation model; and perform separation processing on the time-frequency features based on the scale information of each attention unit through at least one of the attention units in the separator to obtain target track information; A decoding module is configured to input the target audio track information into a decoder of the audio track separation model to obtain a target audio signal, The scale information includes the number of sub-speech frames and the number of sub-frequency bands, the number of sub-speech frames is less than or equal to the number of speech frames in the time-frequency feature, and the number of sub-frequency bands is less than or equal to the number of frequency bands in the time-frequency feature.

12. The device according to claim 11, characterized in that The separation module is configured as: Based on the number of speech frames and the number of frequency bands in the time-frequency features, the scale information of the at least one attention unit is determined respectively, wherein the scale information of each attention unit is different.

13. The device according to claim 11, characterized in that The separation module is configured as: According to the scale information corresponding to each of the attention units, the target time-frequency range is divided into sub-time-frequency ranges, wherein the target time-frequency range is formed based on the number of speech frames and the number of frequency bands in the time-frequency features; For any of the attention units, extracting time-frequency correlation information corresponding to the target audio track information from the target time-frequency feature through the attention unit, wherein the target time-frequency feature is a feature corresponding to the sub-time-frequency range corresponding to the attention unit in the time-frequency feature; and The target audio track information is generated based on the time-frequency correlation information obtained by each of the attention units.

14. The device according to claim 13, characterized in that The separation module is configured as: The time-frequency correlation information respectively obtained by each of the attention units is integrated to obtain the time-frequency correlation information corresponding to the target audio track information; The target audio track information is generated based on the time-frequency correlation information.

15. The device according to any one of claims 11 to 13, characterized in that The separator includes a plurality of separators, each separator includes a plurality of attention units, The separation module is configured as follows: For each separator, the time-frequency correlation information respectively obtained by multiple attention units in the separator is fused to obtain the time-frequency correlation information for the separator; The time-frequency correlation information of each separator is merged to obtain target time-frequency correlation information; The target audio track information is generated based on the target time-frequency correlation information.

16. A training device for a track separation model, the track separation model comprising an encoder, a separator and a decoder, the separator comprising at least one attention unit, characterized in that: The device comprises: An acquisition module is configured to acquire a mixed audio signal and a clean target audio track signal, wherein the mixed audio signal includes the clean target audio track signal; The training module is configured as follows: Using the encoder to extract features from the mixed audio signal to obtain time-frequency features of the mixed audio signal; Based on the time-frequency features, determining scale information of at least one of the attention units; By using at least one of the attention units in the separator, the time-frequency features are separated and processed based on the scale information of each of the attention units to obtain target track information; Decoding the target audio track information using the decoder to obtain an estimated target audio track signal; The track separation model is trained according to the pure target track signal and the estimated target track signal to obtain a trained track separation model, The scale information includes the number of sub-speech frames and the number of sub-frequency bands, the number of sub-speech frames is less than or equal to the number of speech frames in the time-frequency feature, and the number of sub-frequency bands is less than or equal to the number of frequency bands in the time-frequency feature.

17. The device according to claim 16, characterized in that The training module is configured as: Based on the number of speech frames and the number of frequency bands in the time-frequency features, the scale information of the at least one attention unit is determined respectively, wherein the scale information of each attention unit is different.

18. The device according to claim 16, characterized in that The training module is configured as: According to the scale information corresponding to each of the attention units, the target time-frequency range is divided into sub-time-frequency ranges, wherein the target time-frequency range is formed based on the number of speech frames and the number of frequency bands in the time-frequency features; For any of the attention units, extracting time-frequency correlation information corresponding to the target audio track information from the target time-frequency feature through the attention unit, wherein the target time-frequency feature is a feature corresponding to the sub-time-frequency range corresponding to the attention unit in the time-frequency feature; and The target audio track information is generated based on the time-frequency correlation information obtained by each of the attention units.

19. The device according to claim 18, characterized in that The training module is configured as: The time-frequency correlation information respectively obtained by each of the attention units is integrated to obtain the time-frequency correlation information corresponding to the target audio track information; The target audio track information is generated based on the time-frequency correlation information.

20. The device according to any one of claims 16 to 18, characterized in that The separator includes a plurality of separators, each separator includes a plurality of attention units, Among them, the training module is configured as: For each separator, the time-frequency correlation information respectively obtained by multiple attention units in the separator is fused to obtain the time-frequency correlation information for the separator; The time-frequency correlation information of each separator is merged to obtain target time-frequency correlation information; The target audio track information is generated based on the target time-frequency correlation information.

21. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the audio track separation method as described in any one of claims 1 to 5 or the audio track separation model training method as described in any one of claims 6 to 10.

22. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one processor, the at least one processor is prompted to perform the audio track separation method as claimed in any one of claims 1 to 5 or the audio track separation model training method as claimed in any one of claims 6 to 10.

23. A computer program product, wherein instructions in the computer program product are executed by at least one processor in an electronic device to perform the audio track separation method according to any one of claims 1 to 5 or the audio track separation model training method according to any one of claims 6 to 10.

Images