Method, apparatus, device and product for separating audio

Abstract

The invention relates to a method, a device, equipment and a product for separating audio. The method includes generating, by an encoder, a time-domain feature and a frequency-domain feature of a human voice audio based on the human voice audio. The method further includes generating, by the network with the attention mechanism, a fused feature based on the time domain feature and the frequency domain feature. Further, the method includes generating, by a decoder, a separated audio based on the fused feature, where the separated audio includes at least one of a human voice dry audio or a human voice reverberation audio. Through the audio separation mode with the attention mechanism, the distinguishing capability of the audio separation model on the human voice and reverberation components can be enhanced, so that the accuracy of separating the human voice and the reverberation from the human voice track is improved. Besides, the separation method based on the audio frequency can also keep the separated reverberation audio frequency or human voice dry sound audio frequency to have the tone quality close to that of the human voice audio frequency while the audio frequency is successfully separated, so that the user experience is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computers, and more particularly to methods, apparatus, devices and products for separating audio. Background Technology

[0002] Music Source Separation (MSS) refers to the process of separating multiple independent audio signals from a mixed audio stream using a series of processing techniques. In the music industry, MSS technology is widely used in music production and editing. It can extract different instrument tracks, such as vocals, drums, and bass, from mixed music, allowing musicians to more precisely adjust and control musical elements.

[0003] Traditional audio source separation methods primarily rely on signal processing techniques, such as filter design and time-frequency analysis. In recent years, deep learning has made significant progress in audio source separation. By training deep neural network models, effective separation of complex audio signals can be achieved. Deep learning methods possess powerful feature extraction and pattern recognition capabilities, enabling them to handle more complex audio environments. Summary of the Invention

[0004] In a first aspect of the embodiments of this disclosure, a method for separating audio is provided. The method includes generating temporal and frequency domain features of human voice audio by an encoder based on the human voice audio. The method further includes generating fused features by a network having an attention mechanism based on the temporal and frequency domain features. Furthermore, the method includes generating separated audio by a decoder based on the fused features, wherein the separated audio includes at least one of dry human voice audio or reverberated human voice audio.

[0005] In a second aspect of the embodiments of this disclosure, an apparatus for separating audio is provided. The apparatus includes a time-frequency domain feature generation module configured to generate time-domain and frequency-domain features of human voice audio from human voice audio by an encoder. The apparatus also includes a fusion feature generation module configured to generate fusion features from the time-domain and frequency-domain features by a network having an attention mechanism. Furthermore, the apparatus includes a separated audio generation module configured to generate separated audio from the fusion features by a decoder, wherein the separated audio includes at least one of human voice dry audio or human voice reverberation audio.

[0006] In a third aspect of the embodiments of this disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement a method for separating audio. The method includes generating temporal and frequency domain features of human voice audio by an encoder based on human voice audio. The method further includes generating fused features by a network having an attention mechanism based on the temporal and frequency domain features. Furthermore, the method includes generating separated audio by a decoder based on the fused features, wherein the separated audio includes at least one of human voice dry audio or human voice reverberation audio.

[0007] In a fourth aspect of embodiments of this disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to implement a method for separating audio. The method includes generating temporal and frequency domain features of human voice audio by an encoder based on the human voice audio. The method further includes generating fused features by a network having an attention mechanism based on the temporal and frequency domain features. Furthermore, the method includes generating separated audio by a decoder based on the fused features, wherein the separated audio includes at least one of human voice dry audio or human voice reverberation audio.

[0008] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which several embodiments of the present disclosure may be implemented is shown;

[0011] Figure 2 A flowchart of a method for separating audio according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A flowchart of an example process for separating audio according to some embodiments of the present disclosure is shown;

[0013] Figure 4A schematic diagram illustrating an example of obtaining fused features via a multi-head attention mechanism network according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A schematic diagram illustrating an example of a network with a multi-head attention mechanism according to some embodiments of the present disclosure is shown;

[0015] Figures 6A-6B A schematic diagram illustrates an example process for obtaining dry audio or reverberant audio based on wet audio according to some embodiments of the present disclosure;

[0016] Figure 7A A schematic diagram illustrating an example process for training an audio separation model based on the loss between the dry audio and the target dry audio, according to some embodiments of the present disclosure;

[0017] Figure 7B A schematic diagram illustrates an example process for training an audio separation model based on a loss between reverberant audio and target reverberant audio, according to some embodiments of the present disclosure;

[0018] Figure 8 A block diagram of an apparatus for separating audio according to some embodiments of the present disclosure is shown; and

[0019] Figure 9 A block diagram of a device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0020] It is understood that all user-related data involved in this technical solution should be obtained and used only after authorization from the user. This means that if it is necessary to use a user's personal information in this technical solution, the user's explicit consent and authorization are required before obtaining this data; otherwise, no related data collection and use will be carried out. It should also be understood that when implementing this technical solution, relevant laws and regulations should be strictly followed in the process of data collection, use, and storage, and necessary technical measures should be taken to protect user data security and ensure the secure use of data.

[0021] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0022] For example, upon receiving a user's proactive request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0023] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0025] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0026] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0027] As mentioned above, extracting different audio tracks from mixed music allows musicians to easily adjust and control musical elements. For example, when covering a song, it's often necessary to apply reverb to the vocals. This is because reverb simulates the natural reflection of sound in a specific environment, thereby increasing the spatial sense and depth of the music. To ensure the cover vocals have the same reverb effect as the original, it's necessary to separate the reverb-enhanced audio from the original song, and then use this reverb-enhanced audio to apply reverb to the cover vocals.

[0028] However, traditional audio source separation techniques, such as filter processing, often lead to sound quality loss when separating dry vocals from reverberation signals, resulting in increased background noise or audio distortion. On the other hand, some related techniques rely on specific assumptions or parameter settings, limiting their flexibility in applying them to different types of audio materials. Furthermore, although some deep learning techniques, using convolutional layers, have made some progress in audio separation, they exhibit significant shortcomings when dealing with long-range dependencies, making it difficult to handle lengthy audio materials.

[0029] To achieve a reverb effect in cover versions of songs that closely resembles the original vocals, this disclosure provides a method for audio separation. An encoder simultaneously extracts temporal and frequency domain features from the vocal audio, and a network with an attention mechanism fuses these features. A decoder then generates the separated audio based on the fused features. This attention-based approach enhances the audio separation model's ability to distinguish between vocals and reverb components, thereby improving the accuracy of separating vocals and reverb from an audio track. Furthermore, this audio-based separation method maintains a similar sound quality to the original vocal audio while successfully separating the reverb or dry vocal audio, thus improving the user experience.

[0030] Figure 1 A schematic diagram of an example environment 100 in which various embodiments of the present disclosure may be implemented is shown. Reference Figure 1 To achieve accurate audio separation, this disclosure does not employ traditional sound source separation techniques. Instead, it utilizes an audio separation model 120 with an attention mechanism to separate the wet audio 110 of human voice. In some embodiments, the wet audio 110 of human voice can be a human voice audio with reverberation separated from the original music. In some embodiments, before feeding the wet audio 110 of human voice into the audio separation model 120, the expression of the wet audio 110 of human voice in the time-frequency domain can be obtained through short-time Fourier transform.

[0031] like Figure 1 As shown, the audio separation model includes an encoder 122, a network 126 with an attention mechanism, and a decoder 124. In some embodiments, the encoder 122 may include a convolutional layer and a two-layer feature processing module. The convolutional layer may be a 2D convolutional layer, used to extract local temporal features, because a 2D convolutional layer can capture changes in the audio signal at different frequencies over a short period of time.

[0032] In some embodiments, to enable the encoder 122 of the audio separation model 120 to better capture the global time and frequency features in the wet audio 110 of human voice, the feature processing module may include a time-frequency convolution inverse block-time distributed fully connected layer (TFC-TDF). The time-distributed fully connected layer (TDF) can be multiple linear layers cascaded together. For a given frequency domain signal, it can obtain the spectrum-spectrum dependencies of the target signal, thus improving the ability of the audio separation model 120 to process long audio materials. In some embodiments, the time-distributed fully connected layer (TDF) can be a sequence of two linear layers. The time-frequency convolution inverse block (TFC) is a specially designed convolution operation that can be used to process features in both time and frequency dimensions simultaneously, further extracting time-frequency domain feature information from the audio, thereby helping the audio separation model 120 to better understand the characteristics and structure of the sound.

[0033] In some embodiments, to abstract higher-level time and frequency domain features and expand the receptive field of the time and frequency domains, the feature processing module may further include a downsampling layer. In some embodiments, during the downsampling process, common information in the time and frequency domain features may also be extracted, thereby enabling the audio separation model 120 to learn more abstract feature representations. In some embodiments, the downsampling layer may also reduce the dimensionality of the time and frequency features through operations such as pooling.

[0034] Continue to refer to Figure 1 To enhance the ability of the audio separation model 120 to process long audio materials, the audio separation model 120 may include a network 126 with an attention mechanism. In some embodiments, the network 126 with the attention mechanism may be a U-Net network with an attention mechanism. Injecting the time and frequency features processed by the encoder 122 into the U-Net network with the attention mechanism can improve the SDR (signal-to-interference ratio) performance of the audio separation model 120 in audio separation.

[0035] In some embodiments, the self-attention layer in the attention mechanism network 126 can use multi-head attention to process the compressed, abstracted, and integrated time- and frequency-domain features obtained by the encoder 122 in parallel, thereby improving the learning ability and processing efficiency of the audio separation model 120. In some embodiments, these features can be first divided into multiple small blocks, so that multi-head self-attention can be applied to each small block, which allows each head to learn different feature representations. In some embodiments, the results of these multi-head processing can be merged, and then the segmented features can be fused through a linear transformation layer to obtain fused time- and frequency-domain features.

[0036] Continue to refer to Figure 1 Once the fused time and frequency domain features are obtained, the decoder 124 can separate the output audio. In some embodiments, the decoder 124 may have an upsampling layer, a time-frequency convolutional inverse block-time distribution fully connected layer (TFC-TDF), and a 2D convolutional layer. In some embodiments, in the audio separation model 120, the decoder 124 can receive features from the corresponding encoding stage of the encoder 122 at each decoding stage, thereby helping the decoder 124 to recover the detailed information of the audio. This ensures that the separated reverberant audio 130 or dry audio 140 of the human voice has a sound quality close to that of the wet audio 110 of the human voice.

[0037] In some embodiments, if the audio separation model 120 is designed to separate reverberant audio, then the reverberant audio 130 of the human voice can be obtained through the audio separation model 120. Conversely, if the audio separation model 120 is designed to separate dry audio, then the dry audio 140 of the human voice can be obtained through the audio separation model 120. Alternatively, by inputting the wet audio 110 of the human voice into the audio separation model 120, both the reverberant audio 130 and the dry audio 140 of the human voice can be obtained simultaneously.

[0038] This attention-based audio separation method enhances the audio separation model's ability to distinguish between vocals and reverb components, thereby improving the accuracy of separating vocals and reverb from vocal tracks. Furthermore, this audio-based separation method can maintain a similar sound quality to the original vocal audio while successfully separating the audio, thus improving the user experience.

[0039] Figure 2 A flowchart of a method 200 for collaboration of multiple warehouse robots according to some embodiments of the present disclosure is shown. Method 200 can be performed by means of separating audio. Method 200 includes blocks 202, 204, and 206.

[0040] like Figure 2 As shown, in box 202, the encoder generates the temporal and frequency domain features of the human voice audio based on the human voice audio. (Reference) Figure 1In some embodiments, the encoder 122 may include a convolutional layer and a two-layer feature processing module. The convolutional layer may be a 2D convolutional layer used to extract local temporal features. In some embodiments, to enable the encoder 122 of the audio separation model 120 to better capture the global temporal and frequency features of the wet audio 110 of human voice, the feature processing module may include a time-frequency convolutional inverse block-temporally distributed fully connected layer (TFC-TDF). The time-distributed fully connected layer (TDF) may be multiple linear layers cascaded together. For a given frequency domain signal, it can obtain the spectrum-spectral dependencies of the target signal, thus improving the ability of the audio separation model 120 to process long audio materials. In some embodiments, the time-distributed fully connected layer (TDF) may be a sequence of two linear layers. The time-frequency convolutional inverse block (TFC) is a specially designed convolutional operation that can be used to process features in both time and frequency dimensions simultaneously, further extracting time-frequency domain feature information from the audio, thereby helping the audio separation model 120 to better understand the characteristics and structure of the sound. In some embodiments, in order to abstract higher-level time and frequency domain features and expand the receptive field of the time and frequency domains, the feature processing module may also include a downsampling layer. In some embodiments, during the downsampling process, common information in the time domain features and frequency domain features may also be extracted, thereby enabling the audio separation model 120 to learn more abstract feature representations.

[0041] In box 204, a network with an attention mechanism generates fused features based on temporal and frequency domain features. (Continue to refer to...) Figure 1 To enhance the audio separation model 120's ability to process long audio materials, the attention mechanism network 126 can be a U-Net network with an attention mechanism. Injecting the time-frequency features processed by the encoder 122 into the U-Net network with the attention mechanism improves the SDR (Signal-to-Interference Ratio) performance of the audio separation model 120 in audio separation. In some embodiments, the self-attention layer in the attention mechanism network 126 can use multi-head attention to process the compressed, abstracted, and integrated time-frequency domain features obtained by the encoder 122 in parallel, thereby improving the learning ability and processing efficiency of the audio separation model 120. In some embodiments, these features can be first divided into multiple small blocks, allowing multi-head self-attention to be applied to each block, enabling each head to learn different feature representations. In some embodiments, the results of these multi-head processing can be merged, and then a linear transformation layer can be used to fuse these segmented features to obtain fused time-frequency domain features.

[0042] In box 206, the decoder generates separate audio based on fusion features, wherein the separate audio includes at least one of dry vocal audio or reverberated vocal audio. (Continue to reference...) Figure 1 Once the fused time and frequency domain features are obtained, the decoder 124 can separate the output audio. In some embodiments, if the audio separation model 120 is designed to separate reverberant audio, then the reverberant audio 130 of the human voice can be obtained through the audio separation model 120. Conversely, if the audio separation model 120 is designed to separate dry audio, then the dry audio 140 of the human voice can be obtained through the audio separation model 120. Alternatively, by inputting the wet audio 110 of the human voice into the audio separation model 120, both the reverberant audio 130 and the dry audio 140 of the human voice can be obtained simultaneously.

[0043] This attention-based audio separation method enhances the audio separation model's ability to distinguish between vocals and reverb components, thereby improving the accuracy of separating vocals and reverb from vocal tracks. Furthermore, this audio-based separation method can maintain a similar sound quality to the original vocal audio while successfully separating the audio, thus improving the user experience.

[0044] Figure 3 A flowchart of an example process 300 for separating audio according to some embodiments of the present disclosure is shown. Reference Figure 3 You can first separate the original audio (i.e., Figure 1 The wet audio 110 of the human voice shown is subjected to a short-time Fourier transform at 302 to obtain the expression of the original audio in the time-frequency domain, thereby facilitating the extraction of more time-frequency domain information about the original audio. After obtaining the expression information of the original audio in the time-frequency domain, the expression information in the time-frequency domain can be sent to the encoder 310 of the audio separation model. In some embodiments, the encoder 122 includes a 2D convolutional layer 311 and two feature processing modules, one feature processing module including a time-frequency convolutional inverse block-temporal distribution fully connected layer 312 and a downsampling layer 313, and the other feature processing module including a time-frequency convolutional inverse block-temporal distribution fully connected layer 314 and a downsampling layer 315.

[0045] In some embodiments, in order to capture the changes of audio signals at different frequencies over a short period of time, local temporal features (i.e., first temporal features) of the wet audio signal of human voice can be extracted first through a 2D convolutional layer 311. In some embodiments, batch normalization and ReLU activation function (or other nonlinear activation function) can be applied after the 2D convolutional layer to ensure the training stability of the audio separation model.

[0046] In some embodiments, after extracting local temporal features, further temporal features (i.e., second temporal features) can be extracted through the time-frequency convolution inverse block (TFC) in the time-distributed fully connected layer 312. In some embodiments, the time-frequency convolution inverse block (TFC) is a specially designed convolution operation that can be used to process features in both time and frequency dimensions simultaneously, enabling further extraction of time-frequency domain feature information in the wet audio of human voice, thereby helping the audio separation model to better understand the characteristics and structure of sound.

[0047] In some embodiments, frequency domain features (i.e., first frequency domain features) can also be extracted via a time-distributed fully connected layer (TDF) in the time-frequency convolution inverse block-time-distributed fully connected layer 312. In some embodiments, the time-distributed fully connected layer (TDF) can be multiple linear layers cascaded together. For a given frequency domain signal of a wet human voice audio, it can obtain the spectrum-to-spectrum dependence of the target signal, thereby increasing the receptive field and improving the ability of the audio separation model to process long audio materials.

[0048] In some embodiments, a downsampling layer 313 can be used to obtain a first downsampling feature. It is understood that this first downsampling feature includes both temporal and frequency domain features. In some embodiments, the first downsampling feature can be a higher-level, lower-dimensional audio feature. Similarly, to further learn the dependencies between temporal and frequency domain features in long audio materials, the first downsampling feature can be input into a time-frequency convolutional inverse block-temporal distribution fully connected layer 314 and a downsampling layer 315 to obtain a second downsampling feature.

[0049] Continue to refer to Figure 3 After obtaining the second downsampled features containing both temporal and frequency domain features, these features can be injected into a network 126 with an attention mechanism to obtain deeply fused features. For example, the audio separation model 120 can learn the input second downsampled features using the multi-head attention layer 322 of the network 126 with an attention mechanism. The following will combine... Figure 4 and Figure 5 To explain. Figure 4 A schematic diagram of example 400 of obtaining fused features via a multi-head attention mechanism network according to some embodiments of the present disclosure is shown. Figure 5 A schematic diagram of an example 500 of a network having a multi-head attention mechanism according to some embodiments of the present disclosure is shown.

[0050] refer to Figure 4 In some embodiments, at 410, the encoder output (i.e., the second downsampled feature) can be divided into multiple small blocks, for example, 16 small blocks. Combined with Figure 5 That is, attention layer 540 has 16 heads 550. Thus, at 420, a multi-head attention mechanism can be used for each small block, allowing each head to learn different representations of the input features. The structure of multi-head attention layer 322 may include a fully connected layer 510 for receiving a query vector (Q). The structure of multi-head attention layer 322 may also include a fully connected layer 520 for receiving a key vector (K). The structure of multi-head attention layer 322 may also include a fully connected layer 530 for receiving a numerical vector (V). It is understood that each small block has its own query vector, key vector, and numerical vector. It is understood that the encoder's output, as input to the multi-head attention mechanism, has been transformed into query vectors, key vectors, and numerical vectors. Self-attention layer 540 can calculate attention weights between each position and other positions and perform a weighted summation of the numerical vectors based on these weights. This process allows the audio separation model to consider other positions in the entire sequence when processing each position, enabling the audio separation model to capture long-range dependencies, i.e., the ability to process long audio materials.

[0051] Combination Figure 4 After applying multi-head attention to each small block, the results of the multi-head attention processing can be merged at step 430, and then fused using a linear layer. Figure 5 The output of the self-attention layer 540 can be fed into another fully connected layer for further processing. In some embodiments, this fully connected layer typically contains two linear transformations and a ReLU activation function. Its purpose is to perform further nonlinear transformations and dimensionality adjustments on the output of the attention layer 540 to generate the final fused features.

[0052] return Figure 3 In some embodiments, residual connections and layer normalization 324 can be added after the multi-head attention layer 322. The residual connections can alleviate the gradient vanishing problem in the network 126 with the attention mechanism. Similarly, layer normalization can also alleviate the gradient vanishing and gradient exploding problems, thereby improving the training stability of the audio separation model.

[0053] Continue to refer to Figure 3To extract higher-level fusion features, the output of the normalized layer can be used as input to a feedforward network (FFN) 326. The feedforward network contains two linear transformations and a non-linear activation function (such as ReLU). The first linear transformation maps the input fusion features to a higher-dimensional space, thus increasing non-linear expressive power, while the second linear transformation maps the output back to the original dimension or the desired output dimension. To further alleviate the gradient vanishing problem and facilitate the audio separation model's learning of deeper network structures, residual connections and layer normalization can be applied again to the higher-level fusion features 328.

[0054] Continue to refer to Figure 3 The decoder 124 can progressively recover the temporal and frequency domain resolution based on the fused features output by the network 126 with an attention mechanism, thereby obtaining the final separated reverberant or dry audio using the short-time Fourier inverse transform 352. In some embodiments, the decoder 124 includes two feature processing modules and a 2D convolutional layer 335, one feature processing module including an upsampling layer 331 and a time-frequency convolutional inverse block-time distribution fully connected layer 332, and the other feature processing module including an upsampling layer 333 and a time-frequency convolutional inverse block-time distribution fully connected layer 334.

[0055] In some embodiments, the first temporal features and the first frequency features can be gradually recovered through an upsampling layer 331 and a time-frequency convolution inverse block-time distribution fully connected layer 332. Similarly, the second temporal features and the second frequency features of the audio can be recovered based on an upsampling layer 333 and a time-frequency convolution inverse block-time distribution fully connected layer 334. Then, a 2D convolutional layer can be used to obtain the time-frequency domain representation of the separated audio, and the separated audio can be obtained using an inverse short-time Fourier transform 352. In some embodiments, the decoder 124 can receive features from the corresponding encoding stage of the encoder 122 through skip connections 340 at each decoding stage, thereby helping the decoder 124 recover the detailed information of the audio, thus maintaining the sound quality of the separated reverberant audio or dry audio of the human voice close to that of the wet audio 110 of the human voice. In some embodiments, the audio separation model can have two decoder output branches, thereby enabling the simultaneous output of the separated reverberant audio and human voice audio. It is understood that the working process of the decoder 124 is the reverse of the working process of the encoder 122, and therefore will not be described further here.

[0056] The following will combine Figures 6A-6B To describe. Figures 6A-6BSchematic diagrams of example processes 600A and 600B for obtaining dry audio or reverberant audio based on wet audio according to some embodiments of the present disclosure are shown. Reference Figure 6A If the dry audio 630A is separated from the wet audio 610A by the audio separation model 620A, then the reverberant audio 640A can be obtained by subtracting the dry audio 630A from the wet audio 610A. Conversely, referencing... Figure 6B If the reverberant audio 630B is separated from the wet audio 610B by the audio separation model 620B, then the dry audio 640B can be obtained by subtracting the reverberant audio 630B from the wet audio 610B. It is understandable that the structure of the dry audio separation model is the same as that of the reverberant audio separation model, and the loss function can also be the same.

[0057] This attention-based audio separation method enhances the audio separation model's ability to distinguish between vocals and reverb components, thereby improving the accuracy of separating vocals and reverb from vocal tracks. Furthermore, this audio-based separation method can maintain a similar sound quality to the original vocal audio while successfully separating the audio, thus improving the user experience.

[0058] Figure 7A A schematic diagram of an example process 700A for training an audio separation model based on a loss between the dry audio and the target dry audio, according to some embodiments of the present disclosure, is shown. (See reference...) Figure 7A To ensure accurate separation performance of the audio separation model 620A, the parameters of the audio separation model 620A can be adjusted by the loss between the dry audio 630A separated from the wet audio 610A and the true target dry audio 650A. In some embodiments, L1 Loss (also known as Mean Absolute Error) can be used to calculate the loss between the separated dry audio 630A and the true target dry audio 650A.

[0059] Figure 7B A schematic diagram of an example process 700B for training an audio separation model based on a loss between reverberant audio and target reverberant audio, according to some embodiments of this disclosure, is shown. (See reference...) Figure 7BTo ensure accurate separation performance of the audio separation model 620B, the parameters of the audio separation model 620B can be adjusted by the loss between the reverberant audio 630B separated from the wet audio 610B and the true target reverberant audio 650B. In some embodiments, L1 Loss (also known as Mean Absolute Error) can also be used to calculate the loss between the separated reverberant audio 630B and the true target reverberant audio 650B.

[0060] Figure 8 A block diagram of an apparatus 800 for separating audio according to some embodiments of the present disclosure is shown. Figure 8 As shown, the device 800 includes a time-frequency domain feature generation module 802, configured to generate time-domain and frequency-domain features of human voice audio from the human voice audio by an encoder. The device 800 also includes a fusion feature generation module 804, configured to generate fusion features from the time-domain and frequency-domain features by a network with an attention mechanism. Furthermore, the device 800 includes a separate audio generation module 806, configured to generate separate audio from the fusion features by a decoder, wherein the separate audio includes at least one of human voice dry audio or human voice reverberation audio.

[0061] Figure 9 A block diagram of a device 900 capable of implementing various embodiments of the present disclosure is shown. The device 900 may be, for example, as shown below. Figure 1 The processing unit of the picking robot 102 shown. Figure 9 As shown, device 900 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 901, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 902 or loaded from storage unit 908 into random access memory (RAM) 903. Various programs and data required for the operation of device 900 can also be stored in RAM 903. CPU / GPU 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904. Although not shown in... Figure 9 As shown, device 900 may also include a coprocessor.

[0062] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0063] The various methods or processes described above can be executed by CPU / GPU 901. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by CPU / GPU 901, one or more steps or actions in the methods or processes described above can be performed.

[0064] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0065] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0066] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0067] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0068] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0069] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0070] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0071] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0072] The following are some example implementations of this disclosure.

[0073] Example 1. A method for separating audio, comprising:

[0074] The encoder generates the time-domain and frequency-domain features of the human voice audio based on the human voice audio.

[0075] A network with an attention mechanism generates fused features based on the time-domain and frequency-domain features; and a decoder generates separate audio based on the fused features, wherein the separate audio includes at least one of dry human voice audio or reverberant human voice audio.

[0076] Example 2. According to the method described in Example 1, the encoder includes a convolutional layer and two feature processing modules. The feature processing modules include a time-frequency convolutional inverse block-temporal distribution fully connected layer and a downsampling layer. The encoder generates the temporal and frequency domain features of the human voice audio based on the human voice audio, including:

[0077] Based on the human voice audio, a first downsampled feature is obtained through a first feature processing module. The first downsampled feature includes a first downsampled time-domain feature and a first downsampled frequency-domain feature.

[0078] Example 3. The method according to any one of Examples 1-2, wherein obtaining the first downsampled feature based on the first feature processing module includes:

[0079] Based on the human voice audio, the temporal features of the human voice audio are extracted through the convolutional layer of the first feature processing module to obtain the first temporal features; and

[0080] Based on the first time-domain features, the second time-domain features are extracted through the time-frequency convolution inverse block of the first feature processing module.

[0081] Example 4. The method according to any one of Examples 1-3 further includes:

[0082] Based on the human voice audio, the frequency domain features of the human voice audio are determined through the time distribution fully connected layer of the first feature processing module. The time distribution fully connected layer includes multiple linear layers.

[0083] Example 5. The method according to any one of Examples 1-4 further includes:

[0084] Based on the second time-domain feature and the frequency-domain feature of the human voice audio, the first downsampled feature is obtained through the downsampling layer of the first feature processing module.

[0085] Example 6. According to any one of Examples 1-5, where the number of channels in the downsampling layer of the first feature processing module is different from the number of channels in the downsampling layer of the second feature processing module, the method further includes:

[0086] Based on the first downsampling time-domain feature and the first downsampling frequency-domain feature, the second downsampling feature is obtained through the second feature processing module. The second downsampling feature includes the second downsampling time-domain feature and the second downsampling frequency-domain feature.

[0087] Example 7. The method according to any one of Examples 1-6, wherein generating fused features by a network with an attention mechanism based on the temporal features and the frequency domain features includes:

[0088] Based on the second downsampling feature, the fused feature is obtained through the network with the attention mechanism.

[0089] Example 8. The method according to any one of Examples 1-7, wherein obtaining the fused features based on the second downsampling features through the network with the attention mechanism includes:

[0090] The second downsampled feature is divided into multiple smaller downsampled feature blocks;

[0091] Multi-head attention is applied in parallel to the downsampled features of each small block to obtain the multi-head self-attention result of the downsampled features of each small block; and

[0092] The fused features are obtained by linearly transforming the results of multi-head self-attention of the downsampled features of each small block into a fused form.

[0093] Example 9. According to any one of Examples 1-8, the decoder includes two feature transformation modules and a convolutional layer, the feature transformation module including an upsampling layer and a time-frequency convolutional inverse block-temporal distribution fully connected layer, wherein the decoder generates separated audio based on the fused features, comprising:

[0094] Based on the fusion features, the first frequency domain features and the first time domain features of the separated audio are obtained through the first feature transformation module.

[0095] Based on the first frequency domain features and the first time domain features of the separated audio, a second feature transformation module is used to obtain the second frequency domain features and the second time domain features of the separated audio; and

[0096] The separated audio is obtained through the convolutional layer based on the second frequency domain features and the second time domain features of the separated audio.

[0097] Example 10. The method according to any one of Examples 1-9 further includes:

[0098] An audio separation model is trained based on the loss between the separated audio and the real target separated audio. The audio separation model includes the encoder, the network with an attention mechanism, and the decoder.

[0099] Example 11. The method according to any one of Examples 1-10 further includes:

[0100] The human voice audio is acquired and a short-time Fourier transform is performed on the human voice audio to obtain the expression of the human voice audio in the time-frequency domain.

[0101] Example 12. The method according to any one of Examples 1-11 further includes:

[0102] In response to the separated audio being the human voice reverberation audio, the human voice dry audio is determined based on the human voice audio and the human voice reverberation audio; or

[0103] In response to the separated audio being the human voice dry audio, the human voice reverberation audio is determined based on the human voice audio and the human voice dry audio.

[0104] Example 13. An apparatus for separating audio, comprising:

[0105] The time-frequency domain feature generation module is configured to generate time-domain and frequency-domain features of the human voice audio based on the human voice audio by the encoder;

[0106] The fusion feature generation module is configured to generate fusion features based on the time-domain features and frequency-domain features using a network with an attention mechanism; and

[0107] The separate audio generation module is configured to generate separate audio by the decoder based on the fusion features, the separate audio including at least one of dry human voice audio or reverberant human voice audio.

[0108] Example 14. According to the apparatus of Example 13, the encoder includes a convolutional layer and a two-layer feature processing module, the feature processing module including a time-frequency convolutional inverse block-temporal distribution fully connected layer and a downsampling layer, wherein the time-frequency domain feature generation module includes:

[0109] The first obtaining module is configured to obtain a first downsampled feature based on the human voice audio through a first feature processing module. The first downsampled feature includes a first downsampled time-domain feature and a first downsampled frequency-domain feature.

[0110] Example 15. The apparatus according to any one of Examples 13-14, wherein the first obtaining module comprises:

[0111] The second obtaining module is configured to extract temporal features of the human voice audio through the convolutional layer of the first feature processing module, based on the human voice audio, to obtain first temporal features; and

[0112] The first extraction module is configured to extract the second time-domain features based on the first time-domain features and through the time-frequency convolution inverse block of the first feature processing module.

[0113] Example 16. The apparatus according to any one of Examples 13-15 further includes:

[0114] The first determining module is configured to determine the frequency domain features of the human voice audio based on the human voice audio through the time distribution fully connected layer of the first feature processing module, wherein the time distribution fully connected layer includes multiple linear layers.

[0115] Example 17. The apparatus according to any one of Examples 13-16 further includes:

[0116] The third obtaining module is configured to obtain the first downsampled feature based on the second time-domain feature and the frequency-domain feature of the human voice audio through the downsampling layer of the first feature processing module.

[0117] Example 18. The apparatus according to any one of Examples 13-17, wherein the number of channels of the downsampling layer of the first feature processing module is different from the number of channels of the downsampling layer of the second feature processing module, further comprising:

[0118] The fourth obtaining module is configured to obtain a second downsampling feature based on the first downsampling time-domain feature and the first downsampling frequency-domain feature through the second feature processing module. The second downsampling feature includes a second downsampling time-domain feature and a second downsampling frequency-domain feature.

[0119] Example 19. The apparatus according to any one of Examples 13-18, wherein the fusion feature generation module comprises:

[0120] The fifth module is configured to obtain the fused features based on the second downsampled features through the network with the attention mechanism.

[0121] Example 20. The apparatus according to any one of Examples 13-19, wherein the fifth obtaining module comprises:

[0122] The segmentation module is configured to segment the second downsampled feature into multiple smaller downsampled feature blocks;

[0123] The sixth module is configured to apply multi-head attention in parallel to the downsampled features of each block to obtain the result of multi-head self-attention for the downsampled features of each block; and

[0124] The seventh module is configured to obtain the fused features by linearly transforming the results of multi-head self-attention of the downsampled features of each small block into a fused feature.

[0125] Example 21. The apparatus according to any one of Examples 13-20, wherein the decoder comprises two feature transformation modules and a convolutional layer, the feature transformation module comprising an upsampling layer and a time-frequency convolutional inverse block-temporal distribution fully connected layer, wherein the separate audio generation module comprises:

[0126] The eighth obtaining module is configured to obtain the first frequency domain feature and the first time domain feature of the separated audio based on the fusion feature through the first feature transformation module;

[0127] The ninth obtaining module is configured to obtain the second frequency domain features and the second time domain features of the separated audio through a second feature transformation module, based on the first frequency domain features and the first time domain features of the separated audio; and

[0128] The tenth module is configured to obtain the separated audio through the convolutional layer based on the second frequency domain features and the second time domain features of the separated audio.

[0129] Example 22. The apparatus according to any one of Examples 13-21 further includes:

[0130] The training module is configured to train an audio separation model based on the loss between the separated audio and the real target separated audio, the audio separation model including the encoder, the network with an attention mechanism, and the decoder.

[0131] Example 23. The apparatus according to any one of Examples 13-22 further includes:

[0132] The eleventh module is configured to acquire human voice audio and perform a short-time Fourier transform on the human voice audio to obtain the expression of the human voice audio in the time-frequency domain.

[0133] Example 24. The apparatus according to any one of Examples 13-23 further includes:

[0134] The second determining module is configured to, in response to the separated audio being the human voice reverberation audio, determine the human voice dry audio based on the human voice audio and the human voice reverberation audio; or

[0135] The third determining module is configured to determine the human voice reverberation audio based on the human voice audio and the human voice dry audio in response to the separated audio being the human voice dry audio.

[0136] Example 25. An electronic device comprising:

[0137] Processor; and

[0138] A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform actions, the actions including:

[0139] The encoder generates the time-domain and frequency-domain features of the human voice audio based on the human voice audio.

[0140] A network with an attention mechanism generates fused features based on the time-domain and frequency-domain features; and a decoder generates separate audio based on the fused features, wherein the separate audio includes at least one of dry human voice audio or reverberant human voice audio.

[0141] Example 26. The electronic device according to Example 25 further includes:

[0142] In response to detecting movement of a user on a first image on the canvas, determine whether the first image is aligned with the second image in the vertical and / or horizontal directions at the image edges; and

[0143] In response to the detection of the alignment, a first auxiliary reference line is displayed in the vertical direction and / or the horizontal direction of the image edge.

[0144] Example 27. An electronic device according to any one of Examples 25-26, wherein the encoder includes a convolutional layer and a two-layer feature processing module, the feature processing module including a time-frequency convolutional inverse block-time distribution fully connected layer and a downsampling layer, wherein the encoder generates the time-domain features and frequency-domain features of the human voice audio based on the human voice audio, including:

[0145] Based on the human voice audio, a first downsampled feature is obtained through a first feature processing module. The first downsampled feature includes a first downsampled time-domain feature and a first downsampled frequency-domain feature.

[0146] Example 28. An electronic device according to any one of Examples 25-27, wherein obtaining the first downsampled feature based on the first feature processing module includes:

[0147] Based on the human voice audio, the temporal features of the human voice audio are extracted through the convolutional layer of the first feature processing module to obtain the first temporal features; and

[0148] Based on the first time-domain features, the second time-domain features are extracted through the time-frequency convolution inverse block of the first feature processing module.

[0149] Example 29. The electronic device according to any one of Examples 25-28 further includes:

[0150] Based on the second time-domain feature and the frequency-domain feature of the human voice audio, the first downsampled feature is obtained through the downsampling layer of the first feature processing module.

[0151] Example 30. An electronic device according to any one of Examples 25-29, wherein the number of channels of the downsampling layer of the first feature processing module is different from that of the downsampling layer of the second feature processing module, further comprising:

[0152] Based on the first downsampling time-domain feature and the first downsampling frequency-domain feature, the second downsampling feature is obtained through the second feature processing module. The second downsampling feature includes the second downsampling time-domain feature and the second downsampling frequency-domain feature.

[0153] Example 31. An electronic device according to any one of Examples 25-30, wherein the generation of fused features by a network having an attention mechanism based on the time-domain features and the frequency-domain features includes:

[0154] Based on the second downsampling feature, the fused feature is obtained through the network with the attention mechanism.

[0155] Example 32. An electronic device according to any one of Examples 25-31, wherein obtaining the fused features based on the second downsampling features through the network with the attention mechanism includes:

[0156] The second downsampled feature is divided into multiple smaller downsampled feature blocks;

[0157] Multi-head attention is applied in parallel to the downsampled features of each small block to obtain the multi-head self-attention result of the downsampled features of each small block; and

[0158] The fused features are obtained by linearly transforming the results of multi-head self-attention of the downsampled features of each small block into a fused form.

[0159] Example 33. An electronic device according to any one of Examples 25-32, wherein the decoder includes two feature transformation modules and a convolutional layer, the feature transformation module including an upsampling layer and a time-frequency convolutional inverse block-temporal distribution fully connected layer, wherein the decoder generates separate audio based on the fused features, comprising:

[0160] Based on the fusion features, the first frequency domain features and the first time domain features of the separated audio are obtained through the first feature transformation module.

[0161] Based on the first frequency domain features and the first time domain features of the separated audio, a second feature transformation module is used to obtain the second frequency domain features and the second time domain features of the separated audio; and

[0162] The separated audio is obtained through the convolutional layer based on the second frequency domain features and the second time domain features of the separated audio.

[0163] Example 34. The electronic device according to any one of Examples 25-33 further includes:

[0164] An audio separation model is trained based on the loss between the separated audio and the real target separated audio. The audio separation model includes the encoder, the network with an attention mechanism, and the decoder.

[0165] Example 35. The electronic device according to any one of Examples 25-34 further includes:

[0166] The human voice audio is acquired and a short-time Fourier transform is performed on the human voice audio to obtain the expression of the human voice audio in the time-frequency domain.

[0167] Example 36. The electronic device according to any one of Examples 25-35 further includes:

[0168] In response to the separated audio being the human voice reverberation audio, the human voice dry audio is determined based on the human voice audio and the human voice reverberation audio; or

[0169] In response to the separated audio being the human voice dry audio, the human voice reverberation audio is determined based on the human voice audio and the human voice dry audio.

[0170] Example 37. A computer-readable storage medium having stored thereon computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of Examples 1 to 12.

[0171] Example 38. A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of Examples 1 to 12.

[0172] Although this disclosure has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for separating audio, comprising: The encoder generates the time-domain and frequency-domain features of the human voice audio based on the human voice audio. A network with an attention mechanism generates fused features based on the time-domain and frequency-domain features; as well as The decoder generates separate audio based on the fusion features, the separate audio including at least one of dry human voice audio or reverberated human voice audio.

2. The method according to claim 1, wherein the encoder comprises a convolutional layer and two feature processing modules, the feature processing modules comprising a time-frequency convolutional inverse block-temporal distribution fully connected layer and a downsampling layer, wherein the encoder generates the temporal and frequency domain features of the human voice audio based on the human voice audio, including: Based on the human voice audio, a first downsampled feature is obtained through a first feature processing module. The first downsampled feature includes a first downsampled time-domain feature and a first downsampled frequency-domain feature.

3. The method according to claim 2, wherein obtaining the first downsampled feature based on the first feature processing module includes: Based on the human voice audio, the temporal features of the human voice audio are extracted through the convolutional layer of the first feature processing module to obtain the first temporal features; as well as Based on the first time-domain features, the second time-domain features are extracted through the time-frequency convolution inverse block of the first feature processing module.

4. The method according to claim 3, further comprising: Based on the human voice audio, the frequency domain features of the human voice audio are determined through the time distribution fully connected layer of the first feature processing module. The time distribution fully connected layer includes multiple linear layers.

5. The method according to claim 4, further comprising: Based on the second time-domain feature and the frequency-domain feature of the human voice audio, the first downsampled feature is obtained through the downsampling layer of the first feature processing module.

6. The method according to claim 5, wherein the number of channels in the downsampling layer of the first feature processing module is different from the number of channels in the downsampling layer of the second feature processing module, further comprising: Based on the first downsampling time-domain feature and the first downsampling frequency-domain feature, the second downsampling feature is obtained through the second feature processing module. The second downsampling feature includes the second downsampling time-domain feature and the second downsampling frequency-domain feature.

7. The method of claim 6, wherein generating fused features by a network with an attention mechanism based on the temporal features and frequency domain features comprises: Based on the second downsampling feature, the fused feature is obtained through the network with the attention mechanism.

8. The method of claim 7, wherein obtaining the fused features based on the second downsampling features through the network with the attention mechanism comprises: The second downsampled feature is divided into multiple smaller downsampled feature blocks; Multi-head attention is applied in parallel to the downsampled features of each small block to obtain the result of multi-head self-attention of the downsampled features of each small block; as well as The fused features are obtained by linearly transforming the results of multi-head self-attention of the downsampled features of each small block into a fused form.

9. The method according to claim 8, wherein the decoder comprises two feature transformation modules and a convolutional layer, the feature transformation module comprising an upsampling layer and a time-frequency convolutional inverse block-temporal distribution fully connected layer, wherein the decoder generates separated audio based on the fused features, comprising: Based on the fusion features, the first frequency domain features and the first time domain features of the separated audio are obtained through the first feature transformation module. Based on the first frequency domain features and the first time domain features of the separated audio, the second frequency domain features and the second time domain features of the separated audio are obtained through the second feature transformation module. as well as The separated audio is obtained through the convolutional layer based on the second frequency domain features and the second time domain features of the separated audio.

10. The method of claim 9, further comprising: An audio separation model is trained based on the loss between the separated audio and the real target separated audio. The audio separation model includes the encoder, the network with an attention mechanism, and the decoder.

11. The method according to claim 1, further comprising: The human voice audio is acquired and a short-time Fourier transform is performed on the human voice audio to obtain the expression of the human voice audio in the time-frequency domain.

12. The method according to claim 1, further comprising: In response to the separated audio being the human voice reverberation audio, the human voice dry audio is determined based on the human voice audio and the human voice reverberation audio; or In response to the separated audio being the human voice dry audio, the human voice reverberation audio is determined based on the human voice audio and the human voice dry audio.

13. An apparatus for separating audio signals, comprising: The time-frequency domain feature generation module is configured to generate time-domain and frequency-domain features of the human voice audio based on the human voice audio by the encoder; The fusion feature generation module is configured to generate fusion features based on the time-domain features and frequency-domain features by a network with an attention mechanism; as well as The separate audio generation module is configured to generate separate audio by the decoder based on the fusion features, the separate audio including at least one of dry human voice audio or reverberant human voice audio.

14. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1 to 12.

15. A computer program product tangibly stored on a non-transitory computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of claims 1 to 12.

Images