A method, apparatus, device and storage medium for separating an audio signal

Abstract

The application discloses a separation method and device of an audio signal, electronic equipment and a storage medium. The method comprises the following steps: determining the time domain feature code and the frequency domain feature code of the audio signal to be separated based on a feature encoder; determining the time domain interaction feature and the frequency domain interaction feature based on a feature interactor according to the time domain feature code and the frequency domain feature code; wherein the feature interactor comprises a cross-domain interaction unit comprising a recursive gate convolution structure with shared weights; determining the time domain prediction signal according to the time domain interaction feature and the time domain feature code based on a feature decoder, and determining the frequency domain prediction signal according to the frequency domain interaction feature and the frequency domain feature code; and determining the separation result of the audio signal to be separated according to the time domain prediction signal and the frequency domain prediction signal. The technical scheme solves the problems of limited integration information capacity, large calculation amount and high calculation complexity of a mixed domain separation model, can reduce the calculation complexity and the calculation amount while ensuring the separation effect.

Description

Technical Field

[0001] This invention relates to the field of audio processing technology, and in particular to a method, apparatus, electronic device, and storage medium for separating audio signals. Background Technology

[0002] Currently, audio separation methods based on deep learning for multiple signal sources mainly include three categories: (1) time-domain separation methods that primarily learn the temporal variation characteristics of sound; (2) frequency-domain separation methods that learn the pattern characteristics of sound in the frequency domain; and (3) hybrid-domain separation methods that combine the characteristics of the above two types of audio separation. Among them, the separation model based on temporal feature coding takes the temporal waveform of the audio signal to be separated as input and directly learns the temporal feature coding of sound from the temporal waveform to adapt to non-steady-state sound, avoiding the influence of frame length on the temporal resolution of the signal, thereby generating a more natural separated signal. The frequency-domain separation method takes the spectrogram as input, and the separation model decomposes the audio signal to be separated into different frequency bands. It extracts frequency-domain feature coding by combining prior knowledge to improve the modeling ability of the separation model in each frequency band, thereby improving the audio separation effect. In order to make up for the artifact problem caused by the above two types of separation methods on different signal sources, the hybrid-domain model introduces both frequency domain information and temporal domain information into the model so that the separation model can determine the most suitable domain information for each signal source.

[0003] Hybrid domain separation methods typically utilize the Transformer structure for audio signal separation. The Transformer structure enables the separation model to adaptively model and perform long-distance dependency modeling of features. The spatial modeling mechanism of the Self-Attention structure within the Transformer structure demonstrates its ability to integrate information over long sequences. However, the ability of the Self-Attention structure to integrate information over long sequences is achieved by calculating the correlation between two features through dot products, i.e., second-order spatial interaction modeling. Second-order spatial interaction modeling has limited information integration capabilities and suffers from excessive computational cost and complexity. Therefore, an audio signal separation method with strong information integration capabilities and low computational cost and complexity is needed to improve the separation effect of signal sources. Summary of the Invention

[0004] This invention provides an audio signal separation method, apparatus, electronic device, and storage medium to solve the problems of limited information integration capability, large computational load, and high computational complexity of mixed domain separation models. It can reduce computational complexity and computational load while ensuring separation effect.

[0005] According to one aspect of the present invention, a method for separating audio signals is provided, the method comprising:

[0006] Based on the feature encoder, the time-domain feature code and frequency-domain feature code of the audio signal to be separated are determined;

[0007] Based on time-domain feature encoding and frequency-domain feature encoding, time-domain interactive features and frequency-domain interactive features are determined using a feature interactor; wherein, the feature interactor includes at least one cross-domain interactive unit, and each cross-domain interactive unit includes at least one recursive gated convolutional structure with shared weights;

[0008] Based on the feature decoder, the time-domain prediction signal is determined according to the time-domain interaction features and time-domain feature coding, and the frequency-domain prediction signal is determined according to the frequency-domain interaction features and frequency-domain feature coding.

[0009] Based on the time-domain prediction signal and the frequency-domain prediction signal, the separation result of the audio signal to be separated is determined.

[0010] According to another aspect of the present invention, an audio signal separation device is provided, the device comprising:

[0011] The feature coding determination module is used to determine the time-domain feature coding and frequency-domain feature coding of the audio signal to be separated based on the feature encoder.

[0012] An interaction feature determination module is used to determine time-domain interaction features and frequency-domain interaction features based on a feature interactor, according to time-domain feature encoding and frequency-domain feature encoding; wherein, the feature interactor includes at least one cross-domain interaction unit, and each cross-domain interaction unit includes at least one recursive gated convolutional structure with shared weights;

[0013] The prediction signal determination module is used to determine the time-domain prediction signal based on the feature decoder, according to the time-domain interaction features and the time-domain feature encoding, and to determine the frequency-domain prediction signal based on the frequency-domain interaction features and the frequency-domain feature encoding.

[0014] The separation result determination module is used to determine the separation result of the audio signal to be separated based on the time-domain prediction signal and the frequency-domain prediction signal.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the audio signal separation method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the audio signal separation method according to any embodiment of the present invention.

[0020] The technical solution of this invention determines the temporal and frequency-domain feature codes of the audio signal to be separated through a feature encoder; based on the temporal and frequency-domain feature codes, and using a feature interactor, it determines temporal and frequency-domain interactive features; wherein the feature interactor includes at least one cross-domain interactive unit, and each cross-domain interactive unit includes at least one recursively gated convolutional structure with shared weights; based on a feature decoder, it determines a temporal prediction signal according to the temporal interactive features and temporal feature codes, and determines a frequency-domain prediction signal according to the frequency-domain interactive features and frequency-domain feature codes; and determines the separation result of the audio signal to be separated based on the temporal and frequency-domain prediction signals. This solution solves the problems of limited information integration capability, large computational load, and high computational complexity of mixed-domain separation models, and can reduce computational complexity and computational load while ensuring separation effect.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of an audio signal separation method according to Embodiment 1 of the present invention;

[0024] Figure 2A This is a flowchart of an audio signal separation method according to Embodiment 2 of the present invention;

[0025] Figure 2B This is a schematic diagram of a separation model provided in Embodiment 2 of the present invention;

[0026] Figure 2C This is a schematic diagram of the structure of a feature encoder according to Embodiment 2 of the present invention;

[0027] Figure 2D This is a schematic diagram of a recursive gated convolution structure provided in Embodiment 2 of the present invention;

[0028] Figure 2E This is a schematic diagram of the structure of a feature interactor provided in Embodiment 2 of the present invention;

[0029] Figure 2F This is a schematic diagram of a feature decoder according to Embodiment 2 of the present invention;

[0030] Figure 3 This is a schematic diagram of the structure of an audio signal separation device according to Embodiment 3 of the present invention;

[0031] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the audio signal separation method of the present invention. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with the relevant provisions of national laws and regulations.

[0034] Example 1

[0035] Figure 1 This is a flowchart illustrating an audio signal separation method according to Embodiment 1 of the present invention. This embodiment is applicable to audio signal separation scenarios such as mixed speech and multi-source music. The method can be executed by an audio signal separation device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0036] S110. Based on the feature encoder, determine the time-domain feature code and frequency-domain feature code of the audio signal to be separated.

[0037] This solution can be executed by electronic devices such as computers and servers. The electronic devices can pre-acquire the audio signal to be separated. This audio signal can be a mixed audio signal containing multiple sound sources. The electronic devices can use the audio signal to be separated as input data and input it into a pre-trained separation model to achieve audio signal separation. The separation model can be built based on deep learning algorithms, such as convolutional neural networks. The separation model can include units such as feature encoders, feature interactors, and feature decoders.

[0038] Based on a feature encoder, an electronic device can extract the time-domain and frequency-domain features of the audio signal to be separated, resulting in time-domain feature encoding and frequency-domain feature encoding. Specifically, the feature encoder can include two feature encoding branches, used to extract the time-domain and frequency-domain features of the audio signal to be separated, respectively. As part of the separation model, the feature encoder is obtained through model training, and therefore has the ability to extract features from the audio signal to be separated in both the time and frequency domains.

[0039] S120. Based on the time-domain feature coding and frequency-domain feature coding, determine the time-domain interaction features and frequency-domain interaction features using the feature interactor.

[0040] After obtaining the time-domain feature code and the frequency-domain feature code, the electronic device can input the time-domain feature code and the frequency-domain feature code into the feature interaction device in sequence. The feature interaction device can extract the interaction features of the time-domain feature code and the frequency-domain feature code in sequence and output the time-domain interaction features and the frequency-domain interaction features.

[0041] The feature interactor may include at least one cross-domain interaction unit, and each cross-domain interaction unit includes at least one recursively gated convolutional structure with shared weights. The recursively gated convolutional structure with shared weights is trained simultaneously using both temporal and frequency domain feature encodings of the training data. Therefore, the cross-domain interaction unit can achieve cross-domain interaction of features in both the temporal and frequency domains, thereby avoiding artifact problems caused by single-dimensional features. Specifically, artifact problems include the sound hole problem caused by the higher sensitivity of frequency domain feature-based audio signal separation to the phase of bass signals, and the impact of latent noise on audio signal separation based on temporal features.

[0042] It should be noted that both time-domain feature coding and frequency-domain feature coding are used as input data for the feature interactor. The input order can be either time-domain feature coding first, followed by frequency-domain feature coding, or vice versa. In this embodiment, the input order of time-domain feature coding and frequency-domain feature coding in the feature interactor is not restricted.

[0043] S130. Based on the feature decoder, determine the time-domain prediction signal according to the time-domain interaction features and time-domain feature coding, and determine the frequency-domain prediction signal according to the frequency-domain interaction features and frequency-domain feature coding.

[0044] Understandably, a feature decoder can be considered the inverse process of a feature encoder, used to reconstruct time-domain and frequency-domain features, and then output time-domain and frequency-domain predicted signals. Specifically, a feature decoder can include a time-domain decoder and a frequency-domain decoder. The time-domain decoder can determine the time-domain predicted signal based on the time-domain interaction features and time-domain feature encoding output by the feature interactor. Simultaneously, the frequency-domain decoder can determine the frequency-domain predicted signal based on the frequency-domain interaction features and frequency-domain feature encoding output by the feature interactor.

[0045] S140. Based on the time-domain prediction signal and the frequency-domain prediction signal, determine the separation result of the audio signal to be separated.

[0046] After obtaining the time-domain prediction signal and the frequency-domain prediction signal, the electronic device can synthesize the time-domain prediction signal and the frequency-domain prediction signal to determine the separation result of the audio signal to be separated. Specifically, the electronic device can add the time-domain prediction signal and the frequency-domain prediction signal to obtain the separation result, or it can pre-set the matching weights of the time-domain prediction signal and the frequency-domain prediction signal, and use the weighted signal of the time-domain prediction signal and the frequency-domain prediction signal as the separation result.

[0047] This technical solution uses a feature encoder to determine the temporal and frequency domain feature codes of the audio signal to be separated. Based on these codes, and using a feature interactor, it determines temporal and frequency domain interaction features. The feature interactor includes at least one cross-domain interaction unit, and each unit includes at least one recursively gated convolutional structure with shared weights. Based on a feature decoder, it determines a temporal prediction signal based on the temporal interaction features and temporal feature codes, and a frequency prediction signal based on the frequency interaction features and frequency domain feature codes. Finally, it determines the separation result of the audio signal to be separated based on the temporal and frequency prediction signals. This solution addresses the problems of limited information integration capability, high computational load, and high computational complexity in mixed-domain separation models, reducing computational complexity and workload while maintaining separation effectiveness.

[0048] Example 2

[0049] Figure 2A This is a flowchart illustrating an audio signal separation method according to Embodiment 2 of the present invention. This embodiment is a refinement based on the above embodiment. Figure 2A As shown, the method includes:

[0050] S210. Based on the time-domain encoder, determine the time-domain feature code of the audio signal to be separated, and based on the frequency-domain encoder, determine the frequency-domain feature code of the audio signal to be separated.

[0051] Figure 2B This is a schematic diagram of a separation model provided according to Embodiment 2 of the present invention. Figure 2B The feature encoder of the illustrated separation model includes a temporal encoder and a frequency-domain encoder. In this scheme, the temporal encoder may include a feedforward neural network (FFN) structure, at least one convolutional structure, and at least one recursively gated convolutional structure; the frequency-domain encoder may include a Fourier transform structure, a feedforward neural network structure, at least one convolutional structure, and at least one recursively gated convolutional structure. Figure 2C This is a schematic diagram of a feature encoder according to Embodiment 2 of the present invention. The structure of the time-domain encoder can be as follows: Figure 2C As shown, the electronic device inputs the audio signal to be separated into the time-domain encoder, and can obtain the time-domain feature code of the audio signal to be separated.

[0052] The connection relationships between the feedforward neural network structure, convolutional structure, and recursively gated convolutional structure in the frequency domain encoder can be as follows: Figure 2C As shown. In the frequency domain encoder, the audio signal to be separated is input as follows: Figure 2C Prior to the network structure shown, the electronic device could perform a Short-Time Fourier Transform (STFT) on the audio signal to be separated using a Fourier Transform structure to extract frequency domain features containing frequency and phase information. The frequency domain features containing frequency and phase information are then input into a... Figure 2C The network structure shown can be used to obtain the frequency domain feature encoding of the audio signal to be separated. In a specific example, the input data of the time-domain encoder can be represented as... The input data of a frequency domain encoder can be represented as Where B represents the batch size, C represents the feature channel dimension, T represents the time length, and Fq represents the frequency domain feature dimension containing frequency and phase information.

[0053] It should be noted that, regarding Figure 2CThe network structures shown can have different dimensions for the convolutional structures in the time-domain encoder and the frequency-domain encoder. For example, the convolutional structure in the time-domain encoder can be a one-dimensional convolution, while the convolutional structure in the frequency-domain encoder can be a two-dimensional convolution.

[0054] In this scheme, the recursive gated convolution structure is constructed based on preset convolution, linear projection, and element-wise multiplication. Figure 2D This is a schematic diagram of a recursive gated convolution structure provided in Embodiment 2 of the present invention. Figure 2D As shown, a recursive gated convolutional structure can include a combination of convolutional layers, split channels, depthwise convolutional layers, and multilayers (Mullayers) and convolutional layers. The depthwise convolutional layers can increase the receptive field, thus enabling long-distance interaction modeling. The multilayer combination structure can perform linear projections through multiple recursive operations, thereby achieving multi-level spatial interaction modeling.

[0055] S220. Input the temporal feature encoding into the first temporal recursive gated convolutional structure of the current cross-domain interaction unit to determine the first temporal feature of the current cross-domain interaction unit; and input the frequency domain feature encoding into the first frequency domain recursive gated convolutional structure of the current cross-domain interaction unit to determine the first frequency domain feature of the current cross-domain interaction unit.

[0056] Figure 2E This is a schematic diagram of the structure of a feature interactor provided according to Embodiment 2 of the present invention. Figure 2E The cross-domain interaction unit shown includes a first temporal recursive gated convolutional structure, a first frequency domain recursive gated convolutional structure, a second temporal recursive gated convolutional structure, a second frequency domain recursive gated convolutional structure, and a recursive gated convolutional structure with shared weights. The electronic device can adjust the dimensions of temporal feature encodings through structures such as reshape layers and normalization layers, and then input them into the first temporal recursive gated convolutional structure of the current cross-domain interaction unit to determine the first temporal feature of the current cross-domain interaction unit. Simultaneously, the electronic device can adjust the dimensions of frequency domain feature encodings through structures such as reshape layers and normalization layers, and then input them into the first frequency domain recursive gated convolutional structure of the current cross-domain interaction unit to determine the first frequency domain feature of the current cross-domain interaction unit.

[0057] It should be noted that, Figure 2E The network parameters of the recursive gated convolutional structures in each cross-domain interaction unit shown, such as kernel size and number of feature channels, can be the same or different. This embodiment does not limit the network structure parameters of different cross-domain interaction units.

[0058] S230. The first time-domain feature and the first frequency-domain feature are sequentially input into the recursive gated convolution structure of the shared weights of the current cross-domain interaction unit to obtain the second time-domain feature and the second frequency-domain feature of the current cross-domain interaction unit.

[0059] Easy to understand, such as Figure 2E The network structure shown allows the electronic device to first input the first time-domain feature into the recursively gated convolutional structure of the shared weights of the current cross-domain interaction unit to obtain the second time-domain feature, and then input the first frequency-domain feature into the same recursively gated convolutional structure to obtain the second frequency-domain feature. Alternatively, the electronic device can first input the first frequency-domain feature into the recursively gated convolutional structure of the shared weights of the current cross-domain interaction unit to obtain the second frequency-domain feature, and then input the first time-domain feature into the same recursively gated convolutional structure to obtain the second time-domain feature.

[0060] S240. Input the second time-domain feature into the second time-domain recursive gated convolution structure of the current cross-domain interaction unit to determine the third time-domain feature of the current cross-domain interaction unit; and input the second frequency-domain feature into the second frequency-domain recursive gated convolution structure of the current cross-domain interaction unit to determine the third frequency-domain feature of the current cross-domain interaction unit.

[0061] like Figure 2E The network structure shown allows the electronic device to input the second time-domain feature into the second time-domain recursive gated convolutional structure of the current cross-domain interaction unit after obtaining the second time-domain feature and the second frequency-domain feature into the second frequency-domain recursive gated convolutional structure of the current cross-domain interaction unit to determine the third time-domain feature of the current cross-domain interaction unit.

[0062] S250. Based on the first time-domain features and the third time-domain features of the current cross-domain interaction unit, determine the input data of the first time-domain recursive gated convolution structure of the next cross-domain interaction unit; and based on the first frequency-domain features and the third frequency-domain features of the current cross-domain interaction unit, determine the input data of the first frequency-domain recursive gated convolution structure of the next cross-domain interaction unit.

[0063] like Figure 2E As shown, the electronic device can merge the first and third time-domain features of the current cross-domain unit as input data for the first time-domain recursive gated convolutional structure of the next cross-domain interaction unit. Similarly, the electronic device can merge the first and third frequency-domain features of the current cross-domain interaction unit as input data for the first frequency-domain recursive gated convolutional structure of the next cross-domain interaction unit.

[0064] S260. Take the next cross-domain interaction unit as the current cross-domain interaction unit, return to execute the determination of the first time-domain feature of the current cross-domain interaction unit, and the determination of the first frequency-domain feature of the current cross-domain interaction unit, until the input data processing of all cross-domain interaction units in the feature interactor is completed. Based on the first time-domain feature and the third time-domain feature output by the last cross-domain interaction unit in the feature interactor, determine the time-domain interaction feature, and based on the first frequency-domain feature and the third frequency-domain feature output by the last cross-domain interaction unit in the feature interactor, determine the frequency-domain interaction feature.

[0065] After obtaining the input data of the next cross-domain interaction unit, the electronic device can use the next cross-domain interaction unit as the current cross-domain interaction unit, inputting the input data of the first temporal recursive gated convolution structure into the first temporal recursive gated convolution structure of the current cross-domain interaction unit to determine the first temporal feature of the current cross-domain interaction unit; and inputting the input data of the first frequency domain recursive gated convolution structure into the first frequency domain recursive gated convolution structure of the current cross-domain interaction unit to determine the first frequency domain feature of the current cross-domain interaction unit. Return to execute S230-S260 until the input data processing of all cross-domain interaction units in the feature interactor is completed.

[0066] Based on the first and third time-domain features output by the last cross-domain interaction unit in the feature interactor, the time-domain interaction features are determined; and based on the first and third frequency-domain features output by the last cross-domain interaction unit in the feature interactor, the frequency-domain interaction features are determined. For example... Figure 2E As shown, for frequency domain interaction features, electronic devices can adjust the dimensions of frequency domain interaction features through a reshaping layer to facilitate the subsequent output of frequency domain prediction signals.

[0067] S270. Based on the time-domain decoder, a time-domain prediction signal is determined according to the time-domain interaction characteristics and time-domain feature encoding; and based on the frequency-domain decoder, a frequency-domain prediction signal is determined according to the frequency-domain interaction characteristics and frequency-domain feature encoding.

[0068] Similar to the feature encoder, the feature decoder can include a time-domain decoder and a frequency-domain decoder. Figure 2F This is a schematic diagram of a feature decoder according to Embodiment 2 of the present invention. Figure 2F As shown, the decoding layer structures of the temporal decoder and the frequency decoder can be the same. Each decoding layer can include transposed convolutional layers, normalization layers, gated linear units (GLUs), and convolutional layers. It should be noted that while the decoding layer structures of the temporal decoder and the frequency decoder can be the same, the network structure parameters in each decoding layer, such as the convolutional layer dimension and convolutional kernel size, can be the same or different.

[0069] like Figure 2FThe network structure connections in the feature decoder are shown. The electronic device can input time-domain interactive features into the time-domain decoder and determine the time-domain prediction signal based on the output data of the time-domain decoder and the time-domain feature encoding. Simultaneously, frequency-domain interactive features are input into the frequency-domain decoder, and the electronic device can determine the frequency-domain prediction signal based on the output data of the frequency-domain decoder and the frequency-domain feature encoding.

[0070] S280. Based on the time-domain prediction signal and the frequency-domain prediction signal, determine the separation result of the audio signal to be separated.

[0071] In this scheme, optionally, determining the separation result of the audio signal to be separated based on the time-domain prediction signal and the frequency-domain prediction signal includes:

[0072] Perform an inverse Fourier transform on the frequency domain prediction signal to obtain the converted prediction signal;

[0073] The conversion prediction signal is added to the time-domain prediction signal to obtain the separation result of the audio signal to be separated.

[0074] Understandably, for frequency domain prediction signals, electronic devices can use inverse Fourier transforms, such as short-time inverse Fourier transforms, to convert the frequency domain prediction signal into a signal of the same type as the time domain prediction signal. By adding the converted prediction signal to the time domain prediction signal, the electronic device can obtain the separation result of the audio signal to be separated.

[0075] In a specific example, such as Figure 2B The separation model shown can use the scale-invariant signal-to-noise ratio (SI-SNR) as the model's loss function. The formula for calculating the signal-to-noise ratio can be expressed as: in, Indicates the separation result. This indicates the audio signal to be separated.

[0076] This technical solution uses a feature encoder to determine the temporal and frequency domain feature codes of the audio signal to be separated. Based on these codes, and using a feature interactor, it determines temporal and frequency domain interaction features. The feature interactor includes at least one cross-domain interaction unit, and each unit includes at least one recursively gated convolutional structure with shared weights. Based on a feature decoder, it determines a temporal prediction signal based on the temporal interaction features and temporal feature codes, and a frequency prediction signal based on the frequency interaction features and frequency domain feature codes. Finally, it determines the separation result of the audio signal to be separated based on the temporal and frequency prediction signals. This solution addresses the problems of limited information integration capability, high computational load, and high computational complexity in mixed-domain separation models, reducing computational complexity and workload while maintaining separation effectiveness.

[0077] Example 3

[0078] Figure 3 This is a schematic diagram of an audio signal separation device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes:

[0079] The feature encoding determination module 310 is used to determine the time-domain feature encoding and frequency-domain feature encoding of the audio signal to be separated based on the feature encoder;

[0080] The interaction feature determination module 320 is used to determine time-domain interaction features and frequency-domain interaction features based on a feature interactor, according to time-domain feature encoding and frequency-domain feature encoding; wherein, the feature interactor includes at least one cross-domain interaction unit, and each cross-domain interaction unit includes at least one recursive gated convolutional structure with shared weights;

[0081] The prediction signal determination module 330 is used to determine the time-domain prediction signal based on the feature decoder, according to the time-domain interaction features and the time-domain feature encoding, and to determine the frequency-domain prediction signal based on the frequency-domain interaction features and the frequency-domain feature encoding.

[0082] The separation result determination module 340 is used to determine the separation result of the audio signal to be separated based on the time-domain prediction signal and the frequency-domain prediction signal.

[0083] In this scheme, optionally, the feature encoder includes a time-domain encoder and a frequency-domain encoder;

[0084] The feature encoding determination module 310 is specifically used for:

[0085] Based on the time-domain encoder, the time-domain feature code of the audio signal to be separated is determined, and based on the frequency-domain encoder, the frequency-domain feature code of the audio signal to be separated is determined.

[0086] The time-domain encoder includes a feedforward neural network structure, at least one convolutional structure, and at least one recursively gated convolutional structure; the frequency-domain encoder includes a Fourier transform structure, a feedforward neural network structure, at least one convolutional structure, and at least one recursively gated convolutional structure.

[0087] In one feasible scheme, the cross-domain interaction unit further includes a first temporal recursive gated convolution structure, a first frequency domain recursive gated convolution structure, a second temporal recursive gated convolution structure, and a second frequency domain recursive gated convolution structure.

[0088] Based on the above scheme, optionally, the interaction feature determination module 320 is specifically used for:

[0089] The temporal feature encoding is input into the first temporal recursive gated convolutional structure of the current cross-domain interaction unit to determine the first temporal feature of the current cross-domain interaction unit; and the frequency domain feature encoding is input into the first frequency domain recursive gated convolutional structure of the current cross-domain interaction unit to determine the first frequency domain feature of the current cross-domain interaction unit.

[0090] The first time-domain feature and the first frequency-domain feature are sequentially input into the recursive gated convolutional structure of the shared weights of the current cross-domain interaction unit to obtain the second time-domain feature and the second frequency-domain feature of the current cross-domain interaction unit.

[0091] The second temporal feature is input into the second temporal recursive gated convolutional structure of the current cross-domain interaction unit to determine the third temporal feature of the current cross-domain interaction unit; and the second frequency domain feature is input into the second frequency domain recursive gated convolutional structure of the current cross-domain interaction unit to determine the third frequency domain feature of the current cross-domain interaction unit.

[0092] Based on the first and third time-domain features of the current cross-domain interaction unit, determine the input data of the first time-domain recursive gated convolutional structure of the next cross-domain interaction unit; and based on the first and third frequency-domain features of the current cross-domain interaction unit, determine the input data of the first frequency-domain recursive gated convolutional structure of the next cross-domain interaction unit.

[0093] The next cross-domain interaction unit is taken as the current cross-domain interaction unit. The process returns to determine the first time-domain feature and the first frequency-domain feature of the current cross-domain interaction unit. This process continues until the input data processing of all cross-domain interaction units in the feature interactor is completed. Based on the first and third time-domain features output by the last cross-domain interaction unit in the feature interactor, the time-domain interaction feature is determined. Based on the first and third frequency-domain features output by the last cross-domain interaction unit in the feature interactor, the frequency-domain interaction feature is determined.

[0094] In this embodiment, the feature decoder includes a time-domain decoder and a frequency-domain decoder;

[0095] The prediction signal determination module 330 is specifically used for:

[0096] Based on the time-domain decoder, the time-domain prediction signal is determined according to the time-domain interaction features and time-domain feature encoding; and based on the frequency-domain decoder, the frequency-domain prediction signal is determined according to the frequency-domain interaction features and frequency-domain feature encoding.

[0097] Optionally, the separation result determination module 340 is specifically used for:

[0098] Perform an inverse Fourier transform on the frequency domain prediction signal to obtain the converted prediction signal;

[0099] The conversion prediction signal is added to the time-domain prediction signal to obtain the separation result of the audio signal to be separated.

[0100] In a preferred embodiment, the recursive gated convolution structure is constructed based on preset convolution, linear projection, and element-wise multiplication.

[0101] The audio signal separation device provided in this embodiment of the invention can execute the audio signal separation method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0102] Example 4

[0103] Figure 4 A schematic diagram of an electronic device 410 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0104] like Figure 4 As shown, the electronic device 410 includes at least one processor 411 and a memory, such as a read-only memory (ROM) 412 or a random access memory (RAM) 413, communicatively connected to the at least one processor 411. The memory stores computer programs executable by the at least one processor. The processor 411 can perform various appropriate actions and processes based on the computer program stored in the ROM 412 or loaded from storage unit 418 into the RAM 413. The RAM 413 may also store various programs and data required for the operation of the electronic device 410. The processor 411, ROM 412, and RAM 413 are interconnected via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0105] Multiple components in electronic device 410 are connected to I / O interface 415, including: input unit 416, such as keyboard, mouse, etc.; output unit 417, such as various types of displays, speakers, etc.; storage unit 418, such as disk, optical disk, etc.; and communication unit 419, such as network card, modem, wireless transceiver, etc. Communication unit 419 allows electronic device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0106] Processor 411 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 411 performs the various methods and processes described above, such as methods for separating audio signals.

[0107] In some embodiments, the audio signal separation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 410 via ROM 412 and / or communication unit 419. When the computer program is loaded into RAM 413 and executed by processor 411, one or more steps of the audio signal separation method described above may be performed. Alternatively, in other embodiments, processor 411 may be configured to perform the audio signal separation method by any other suitable means (e.g., by means of firmware).

[0108] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0109] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0110] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0111] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0112] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0113] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0114] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0115] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for separating audio signals, characterized in that, The method includes: Based on the feature encoder, the time-domain feature code and frequency-domain feature code of the audio signal to be separated are determined; Based on time-domain feature encoding and frequency-domain feature encoding, time-domain interactive features and frequency-domain interactive features are determined using a feature interactor; wherein, the feature interactor includes at least one cross-domain interactive unit, and each cross-domain interactive unit includes at least one recursive gated convolutional structure with shared weights; Based on the feature decoder, the time-domain prediction signal is determined according to the time-domain interaction features and time-domain feature coding, and the frequency-domain prediction signal is determined according to the frequency-domain interaction features and frequency-domain feature coding. Based on the time-domain prediction signal and the frequency-domain prediction signal, determine the separation result of the audio signal to be separated; The cross-domain interaction unit further includes a first time-domain recursive gated convolution structure, a first frequency-domain recursive gated convolution structure, a second time-domain recursive gated convolution structure, and a second frequency-domain recursive gated convolution structure. The step of determining time-domain interaction features and frequency-domain interaction features based on time-domain feature coding and frequency-domain feature coding, using a feature interactor, includes: The temporal feature encoding is input into the first temporal recursive gated convolutional structure of the current cross-domain interaction unit to determine the first temporal feature of the current cross-domain interaction unit; and the frequency domain feature encoding is input into the first frequency domain recursive gated convolutional structure of the current cross-domain interaction unit to determine the first frequency domain feature of the current cross-domain interaction unit. The first time-domain feature and the first frequency-domain feature are sequentially input into the recursive gated convolutional structure of the shared weights of the current cross-domain interaction unit to obtain the second time-domain feature and the second frequency-domain feature of the current cross-domain interaction unit. The second temporal feature is input into the second temporal recursive gated convolutional structure of the current cross-domain interaction unit to determine the third temporal feature of the current cross-domain interaction unit; and the second frequency domain feature is input into the second frequency domain recursive gated convolutional structure of the current cross-domain interaction unit to determine the third frequency domain feature of the current cross-domain interaction unit. Based on the first and third time-domain features of the current cross-domain interaction unit, determine the input data of the first time-domain recursive gated convolutional structure of the next cross-domain interaction unit; and based on the first and third frequency-domain features of the current cross-domain interaction unit, determine the input data of the first frequency-domain recursive gated convolutional structure of the next cross-domain interaction unit. The next cross-domain interaction unit is taken as the current cross-domain interaction unit. The process returns to determine the first time-domain feature and the first frequency-domain feature of the current cross-domain interaction unit. This process continues until the input data processing of all cross-domain interaction units in the feature interactor is completed. Based on the first and third time-domain features output by the last cross-domain interaction unit in the feature interactor, the time-domain interaction feature is determined. Based on the first and third frequency-domain features output by the last cross-domain interaction unit in the feature interactor, the frequency-domain interaction feature is determined.

2. The method according to claim 1, characterized in that, The feature encoder includes a time-domain encoder and a frequency-domain encoder; The step of determining the time-domain and frequency-domain feature codes of the audio signal to be separated based on the feature encoder includes: Based on the time-domain encoder, the time-domain feature code of the audio signal to be separated is determined, and based on the frequency-domain encoder, the frequency-domain feature code of the audio signal to be separated is determined. The time-domain encoder includes a feedforward neural network structure, at least one convolutional structure, and at least one recursively gated convolutional structure; the frequency-domain encoder includes a Fourier transform structure, a feedforward neural network structure, at least one convolutional structure, and at least one recursively gated convolutional structure.

3. The method according to claim 1, characterized in that, The feature decoder includes a time-domain decoder and a frequency-domain decoder; The feature-based decoder determines the time-domain prediction signal based on time-domain interaction features and time-domain feature encoding, and determines the frequency-domain prediction signal based on frequency-domain interaction features and frequency-domain feature encoding, including: Based on the time-domain decoder, the time-domain prediction signal is determined according to the time-domain interaction features and time-domain feature encoding; and based on the frequency-domain decoder, the frequency-domain prediction signal is determined according to the frequency-domain interaction features and frequency-domain feature encoding.

4. The method according to claim 1, characterized in that, The step of determining the separation result of the audio signal to be separated based on the time-domain prediction signal and the frequency-domain prediction signal includes: Perform an inverse Fourier transform on the frequency domain prediction signal to obtain the converted prediction signal; The conversion prediction signal is added to the time-domain prediction signal to obtain the separation result of the audio signal to be separated.

5. The method according to claim 1, characterized in that, The recursive gated convolution structure is constructed based on preset convolution, linear projection, and element-wise multiplication.

6. An audio signal separation device, characterized in that, The device includes: The feature coding determination module is used to determine the time-domain feature coding and frequency-domain feature coding of the audio signal to be separated based on the feature encoder. An interaction feature determination module is used to determine time-domain interaction features and frequency-domain interaction features based on a feature interactor, according to time-domain feature encoding and frequency-domain feature encoding; wherein, the feature interactor includes at least one cross-domain interaction unit, and each cross-domain interaction unit includes at least one recursive gated convolutional structure with shared weights; The prediction signal determination module is used to determine the time-domain prediction signal based on the feature decoder, according to the time-domain interaction features and the time-domain feature encoding, and to determine the frequency-domain prediction signal based on the frequency-domain interaction features and the frequency-domain feature encoding. The separation result determination module is used to determine the separation result of the audio signal to be separated based on the time-domain prediction signal and the frequency-domain prediction signal; The cross-domain interaction unit further includes a first time-domain recursive gated convolution structure, a first frequency-domain recursive gated convolution structure, a second time-domain recursive gated convolution structure, and a second frequency-domain recursive gated convolution structure. The interaction feature determination module is specifically used for: The temporal feature encoding is input into the first temporal recursive gated convolutional structure of the current cross-domain interaction unit to determine the first temporal feature of the current cross-domain interaction unit; and the frequency domain feature encoding is input into the first frequency domain recursive gated convolutional structure of the current cross-domain interaction unit to determine the first frequency domain feature of the current cross-domain interaction unit. The first time-domain feature and the first frequency-domain feature are sequentially input into the recursive gated convolutional structure of the shared weights of the current cross-domain interaction unit to obtain the second time-domain feature and the second frequency-domain feature of the current cross-domain interaction unit. The second temporal feature is input into the second temporal recursive gated convolutional structure of the current cross-domain interaction unit to determine the third temporal feature of the current cross-domain interaction unit; and the second frequency domain feature is input into the second frequency domain recursive gated convolutional structure of the current cross-domain interaction unit to determine the third frequency domain feature of the current cross-domain interaction unit. Based on the first and third time-domain features of the current cross-domain interaction unit, determine the input data of the first time-domain recursive gated convolutional structure of the next cross-domain interaction unit; and based on the first and third frequency-domain features of the current cross-domain interaction unit, determine the input data of the first frequency-domain recursive gated convolutional structure of the next cross-domain interaction unit. The next cross-domain interaction unit is taken as the current cross-domain interaction unit. The process returns to determine the first time-domain feature and the first frequency-domain feature of the current cross-domain interaction unit. This process continues until the input data processing of all cross-domain interaction units in the feature interactor is completed. Based on the first and third time-domain features output by the last cross-domain interaction unit in the feature interactor, the time-domain interaction feature is determined. Based on the first and third frequency-domain features output by the last cross-domain interaction unit in the feature interactor, the frequency-domain interaction feature is determined.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method for separating audio signals according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method for separating audio signals according to any one of claims 1-5.

Images