Separation method, device, and apparatus for audio signals, storage medium, and program

Abstract

Embodiments of the present disclosure provide a method, apparatus, device, storage medium and program for separating audio signals, the method comprising: determining first amplitude information of a mixed audio signal to be processed, and first phase information of the mixed audio signal; processing the first amplitude information to obtain amplitude difference information and phase difference information between the mixed audio signal and a first audio signal, the first audio signal being a pure audio signal corresponding to a first audio source in the mixed audio signal; and determining the first audio signal according to the first amplitude information, the first phase information, the amplitude difference information and the phase difference information. Through the above process, the audio separation effect can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more particularly to a method, apparatus, device, storage medium, and program for separating audio signals. Background Technology

[0002] Audio signal separation is a technique that separates the pure audio signal of a single sound source from a mixed audio signal. Taking music source separation (MSS) as an example, audio signal separation technology can be used to separate vocals, drum sounds, bass sounds, etc. from a piece of music.

[0003] In related technologies, audio signal separation can be performed in the following way. Taking a first audio source as an example, a source separation model corresponding to the first audio source is pre-trained. The source separation model is used to predict the amplitude of the pure audio signal corresponding to the first audio source based on the mixed audio signal. During audio signal separation, the mixed audio signal is input into the above source separation model to obtain the predicted amplitude. The above predicted amplitude is used as the amplitude of the pure audio signal corresponding to the first audio source, and the phase of the mixed audio signal is used as the phase of the pure audio signal corresponding to the first audio source, thereby obtaining the pure audio signal corresponding to the first audio source.

[0004] However, the inventors discovered that the above-mentioned related technologies have at least the following technical problems: the pure audio signal obtained by the above method is distorted, resulting in poor separation effect. Summary of the Invention

[0005] This disclosure provides an audio signal separation method, apparatus, device, storage medium, and program to improve the audio signal separation effect.

[0006] In a first aspect, embodiments of this disclosure provide a method for separating audio signals, comprising:

[0007] The first amplitude information and the first phase information of the mixed audio signal to be processed are determined, wherein the mixed audio signal is formed by mixing pure audio signals corresponding to multiple sound sources;

[0008] The first amplitude information is processed to obtain amplitude difference information and phase difference information between the mixed audio signal and the first audio signal, wherein the first audio signal is the pure audio signal corresponding to the first sound source in the mixed audio signal;

[0009] The first audio signal is determined based on the first amplitude information, the first phase information, the amplitude difference information, and the phase difference information.

[0010] Secondly, embodiments of this disclosure provide an audio signal separation device, comprising:

[0011] The first determining module is used to determine the first amplitude information of the mixed audio signal to be processed and the first phase information of the mixed audio signal, wherein the mixed audio signal is formed by mixing pure audio signals corresponding to multiple sound sources;

[0012] The processing module is used to process the first amplitude information to obtain amplitude difference information and phase difference information between the mixed audio signal and the first audio signal, wherein the first audio signal is the pure audio signal corresponding to the first sound source in the mixed audio signal;

[0013] The second determining module is used to determine the first audio signal based on the first amplitude information, the first phase information, the amplitude difference information, and the phase difference information.

[0014] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0015] The memory stores computer-executed instructions;

[0016] The processor executes the computer execution instructions to implement the methods described in the first aspect and various possible designs of the first aspect.

[0017] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the methods described in the first aspect and various possible designs of the first aspect.

[0018] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect and various possible designs.

[0019] This disclosure provides an audio signal separation method, apparatus, device, storage medium, and program. The method includes: determining first amplitude information and first phase information of a mixed audio signal to be processed; processing the first amplitude information to obtain amplitude difference information and phase difference information between the mixed audio signal and a first audio signal; wherein the first audio signal is a pure audio signal corresponding to a first sound source in the mixed audio signal; and determining the first audio signal based on the first amplitude information, the first phase information, the amplitude difference information, and the phase difference information. In this process, by processing the first amplitude information, the amplitude difference information and phase difference information between the mixed audio signal and the first audio signal can be predicted separately, ensuring the accuracy of the amplitude and phase information of the first audio signal, thereby avoiding the problem of distortion of the first audio signal and improving the audio separation effect. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A schematic diagram illustrating music source separation provided in an embodiment of this disclosure;

[0022] Figure 2 A schematic diagram of a system architecture provided for an embodiment of this disclosure;

[0023] Figure 3 A flowchart illustrating an audio signal separation method provided in this embodiment of the disclosure;

[0024] Figure 4 A schematic diagram illustrating an audio signal separation process provided in an embodiment of this disclosure;

[0025] Figure 5 A schematic flowchart illustrating another audio signal separation method provided in this disclosure embodiment;

[0026] Figure 6 A schematic diagram illustrating another audio signal separation process provided in an embodiment of this disclosure;

[0027] Figure 7A and Figure 7B A schematic diagram of an additive noise model provided in an embodiment of this disclosure;

[0028] Figure 8A schematic diagram of the distribution of cIRM provided in an embodiment of this disclosure;

[0029] Figure 9A This is a schematic diagram of the structure of a sound source separation model provided in an embodiment of the present disclosure;

[0030] Figure 9B for Figure 9A A schematic diagram of the REB structure in the diagram;

[0031] Figure 9C for Figure 9A A schematic diagram of the structure of RDB in the diagram;

[0032] Figure 9D for Figure 9B and Figure 9C A schematic diagram of the RCB structure in the diagram;

[0033] Figure 10 A schematic diagram of the structure of an audio signal separation device provided in an embodiment of this disclosure;

[0034] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0036] In this embodiment of the disclosure, the sound source is the source of the sound. For example, a piece of music may correspond to multiple sound sources, including but not limited to: vocals, drum sounds, bass sounds, piano sounds, etc.

[0037] This disclosure can be applied to scenarios where a clean audio signal corresponding to a single sound source needs to be separated from a mixed audio signal, such as Music Source Separation (MSS). Music source separation is an important topic in Music Information Retrieval (MIR) and can be used for MIR tasks including, but not limited to, melody extraction, pitch estimation, music transcription, and music mixing. Music source separation also has some direct applications, such as karaoke and music mixing. For ease of understanding, the application scenarios of this disclosure will be described below using MSS as an example.

[0038] Figure 1This is a schematic diagram illustrating music source separation according to an embodiment of the present disclosure. A piece of music is typically obtained by mixing audio signals from different sound sources (e.g., vocals, drums, bass, etc.). In this embodiment, the music can be referred to as a mixed audio signal, and the audio signal generated by each sound source can be referred to as the pure audio signal corresponding to that sound source. See also... Figure 1 When performing music source separation, different sound source separation models can be used to separate the pure audio signal corresponding to each sound source from the music. For example, a vocal separation model can be used to separate vocals from music, a drum sound separation model can be used to separate drum sounds from music, a bass sound separation model can be used to separate bass sounds from music, and so on.

[0039] Figure 2 This is a schematic diagram of a system architecture provided for an embodiment of this disclosure. (As shown...) Figure 2 As shown, the system architecture includes: training equipment and execution equipment.

[0040] The training device can learn and model multiple sets of training samples in the training dataset to obtain a sound source separation model. For example, multiple sets of training samples can be constructed, each set including a mixed audio signal and a clean audio signal corresponding to the first sound source in the mixed audio signal. By learning from multiple sets of training samples, a sound source separation model is obtained, enabling the sound source separation model to separate the clean audio signal of the first sound source.

[0041] The audio source separation model trained by the training device is deployed to the execution device. When audio separation is required, the mixed audio signal to be processed is input to the execution device. The execution device outputs the clean audio signal corresponding to the first audio source. The audio source separation model described above can be used during the processing of the execution device.

[0042] Figure 2 In the system architecture shown, the training device is typically a server. The execution device can be a terminal device or a server. Terminal devices include, but are not limited to, smartphones, platform computers, laptops, smart TVs, smart wearable devices, smart speakers, audio processing devices, etc. It should be noted that the above system architecture is only provided as some possible examples and should not be construed as limiting the embodiments of this disclosure. In some application scenarios, the training device and the execution device can be independent electronic devices. In other application scenarios, the training device and the execution device can be the same electronic device.

[0043] One possible implementation involves audio signal separation in the following manner. Taking a first audio source as an example, a source separation model corresponding to the first audio source is pre-trained. This model is used to predict the amplitude information of the pure audio signal corresponding to the first audio source based on the mixed audio signal. During audio signal separation, the mixed audio signal is input into the aforementioned source separation model to obtain the predicted amplitude information. This predicted amplitude information is then used as the amplitude information of the pure audio signal corresponding to the first audio source, and the phase information of the mixed audio signal is used as the phase information of the pure audio signal corresponding to the first audio source, thereby obtaining the pure audio signal corresponding to the first audio source.

[0044] In another possible implementation, audio signal separation can be performed as follows: Taking a first audio source as an example, a source separation model corresponding to the first audio source is pre-trained. This model is used to predict the amplitude difference information between the mixed audio signal and the pure audio signal corresponding to the first audio source. During audio signal separation, the mixed audio signal is input into the aforementioned source separation model to obtain amplitude difference information. Based on the amplitude information of the mixed audio signal and the aforementioned amplitude difference information, the amplitude information of the pure audio signal corresponding to the first audio source is determined.

[0045] Specifically, an Ideal Ratio Mask (IRM) is used to characterize the aforementioned amplitude difference information. Let IRM be denoted as... The clean audio signal corresponding to the first sound source can be obtained in the following way:

[0046]

[0047] Where |X| represents the amplitude information of the mixed audio signal, This represents the amplitude information of the clean audio signal corresponding to the first sound source. When using an IRM, The value range is [0, 1], which means that the amplitude of the pure audio signal of a single audio source is less than the amplitude of the mixed audio signal.

[0048] Furthermore, the phase information of the mixed audio signal is used as the phase information of the pure audio signal corresponding to the first sound source to obtain the pure audio signal corresponding to the first sound source.

[0049] However, during their research, the inventors discovered that the pure audio signals obtained through the two implementation methods described above suffer from distortion, resulting in poor separation performance. Further analysis revealed that in practical applications, the mixed audio signal is obtained by mixing pure audio signals from multiple audio sources. The phases of these pure audio signals from different sources may vary. In the two implementation methods described above, directly using the phase of the mixed audio signal as the phase of the pure audio signal corresponding to the first audio source leads to inaccurate phase of the pure audio signal from the first source, resulting in distortion and affecting the separation performance.

[0050] This disclosure provides a method, apparatus, device, storage medium, and program for separating audio signals. When separating mixed audio signals, by processing the amplitude information of the mixed audio signals, it is possible not only to predict the amplitude difference between the mixed audio signals and the pure audio signals corresponding to the first sound source, but also to predict the phase difference between the mixed audio signals and the pure audio signals corresponding to the first sound source. Thus, based on the amplitude information, phase information, and the predicted amplitude and phase difference information of the mixed audio signals, the pure audio signal corresponding to the first sound source can be determined.

[0051] In the above process, since the amplitude difference information and phase difference information between the mixed audio signal and the pure audio signal corresponding to the first audio source are predicted at the same time, the accuracy of the amplitude information and phase information of the pure audio signal corresponding to the first audio source is ensured, thereby avoiding the problem of distortion of the pure audio signal corresponding to the first audio source and improving the audio separation effect.

[0052] The technical solutions of this disclosure will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0053] Figure 3 This is a schematic flowchart illustrating an audio signal separation method provided in an embodiment of this disclosure. Figure 3 As shown, the method in this embodiment includes:

[0054] S301: Determine the first amplitude information of the mixed audio signal to be processed and the first phase information of the mixed audio signal, wherein the mixed audio signal is formed by mixing pure audio signals corresponding to multiple sound sources.

[0055] In this embodiment of the disclosure, the mixed audio signal is the audio signal to be separated, and the mixed audio signal includes pure audio signals corresponding to multiple sound sources. For example, the mixed audio signal is a piece of music.

[0056] In a possible implementation, a Fourier transform is performed on the mixed audio signal to be processed to obtain the frequency domain signal corresponding to the mixed audio signal. Based on the frequency domain signal, the amplitude information and phase information of the mixed audio signal are determined. For the sake of brevity in the following description, the amplitude information of the mixed audio signal is referred to as the first amplitude information, and the phase information of the mixed audio signal is referred to as the first phase information.

[0057] In practical applications, the duration of the mixed audio signals to be separated may be relatively long. For example, a piece of music is typically 3-5 minutes long. One possible implementation is to segment the mixed audio signal into multiple segments according to a preset duration (e.g., 1 second, 3 seconds, or 5 seconds). Each segment is treated as the mixed audio signal to be processed, and a short-time Fourier transform is performed on that segment to determine the first amplitude information and the first phase information. Then, the audio signal classification method provided in this embodiment is used for separation processing. Optionally, different segments can be executed in parallel. This can improve the processing efficiency of audio separation.

[0058] S302: Process the first amplitude information to obtain amplitude difference information and phase difference information between the mixed audio signal and the first audio signal, wherein the first audio signal is the pure audio signal corresponding to the first sound source in the mixed audio signal.

[0059] This disclosure describes an embodiment using the example of separating a pure audio signal corresponding to a first audio source from a mixed audio signal. The first audio source can be any one of the multiple audio sources included in the mixed audio signal. For ease of description, in this embodiment, the pure audio signal corresponding to the first audio source in the mixed audio signal is referred to as the first audio source signal.

[0060] In one possible implementation, a pre-trained machine learning model can be used for audio separation processing. In this embodiment, the machine learning model is referred to as a sound source separation model. The sound source separation model can be obtained by learning from multiple sets of training samples. Each set of training samples includes: amplitude information of the sample mixed audio signal, amplitude difference information and phase difference information between the sample mixed audio signal and the sample pure audio signal, wherein the sample pure audio signal is the pure audio signal corresponding to the first sound source in the sample mixed audio signal.

[0061] Figure 4 This is a schematic diagram illustrating an audio signal separation process provided in an embodiment of this disclosure. Figure 4As shown, the first amplitude information (i.e., the amplitude information of the mixed audio signal) is input into the source separation model corresponding to the first audio source. The source separation model processes the mixed audio signal to obtain amplitude difference information and phase difference information. Among them, the amplitude difference information indicates the amplitude difference between the mixed audio signal and the pure audio signal corresponding to the first audio source, and the phase difference information indicates the phase difference between the mixed audio signal and the pure audio signal corresponding to the first audio source.

[0062] It should be noted that, in this embodiment of the disclosure, the difference between A and B should not be limited to the value between A and B. The difference between A and B can be reflected by various relationships. For example, the relationship between A and B can be a multiple relationship, a linear relationship, a non-linear relationship, or any other relationship that can indicate the relationship between A and B.

[0063] In this embodiment, by using the audio source separation model corresponding to the first audio source, not only can the amplitude difference information between the mixed audio signal and the first audio signal be predicted, but also the phase difference information between the mixed audio signal and the first audio signal can be predicted. Compared with the aforementioned implementation method, which directly uses the phase information of the mixed audio signal as the phase information of the first audio signal, the accuracy of the phase information of the first audio signal is improved.

[0064] In this embodiment, the sound source separation model corresponding to the first sound source is pre-trained using machine learning methods. The specific structure and training method of the sound source separation model are not detailed in this embodiment; please refer to the detailed descriptions in subsequent embodiments.

[0065] S303: Determine the first audio signal based on the first amplitude information, the first phase information, the amplitude difference information, and the phase difference information.

[0066] See also Figure 4 Since the source separation model corresponding to the first audio source predicts the amplitude difference information and phase difference information between the mixed audio signal and the first audio signal, the first audio signal can be determined based on the first amplitude information, the first phase information, the amplitude difference information and the phase difference information.

[0067] In a possible implementation, second amplitude information is determined based on the first amplitude information and amplitude difference information; the second amplitude information can be considered as the amplitude information of the first audio signal. Second phase information is determined based on the first phase information and phase difference information; the second phase information can be considered as the phase information of the first audio signal. Thus, the first audio signal can be determined based on the second amplitude information and the second phase information.

[0068] Specifically, the second amplitude information is used as the amplitude information of the first audio signal, and the second phase information is used as the phase information of the first audio signal to obtain the frequency domain signal corresponding to the first audio signal. An inverse Fourier transform is performed on the frequency domain signal corresponding to the first audio signal to obtain the first audio signal.

[0069] The audio signal separation method provided in this embodiment includes: determining first amplitude information and first phase information of the mixed audio signal to be processed; processing the first amplitude information to obtain amplitude difference information and phase difference information between the mixed audio signal and the first audio signal; the first audio signal being the pure audio signal corresponding to the first sound source in the mixed audio signal; and determining the first audio signal based on the first amplitude information, the first phase information, the amplitude difference information, and the phase difference information. In the above process, through the sound source separation model corresponding to the first sound source, the amplitude difference information and phase difference information between the mixed audio signal and the first audio signal can be predicted separately, while ensuring the accuracy of the amplitude and phase information of the first audio signal, thereby avoiding the problem of distortion of the first audio signal and improving the audio separation effect.

[0070] Based on the above embodiments, the audio signal separation method provided in this disclosure will be described in more detail below with reference to a more specific embodiment.

[0071] Figure 5 This is a schematic flowchart of another audio signal separation method provided in an embodiment of the present disclosure.

[0072] like Figure 5 As shown, the method in this embodiment includes:

[0073] S501: Acquire the mixed audio signal to be processed. The mixed audio signal is formed by mixing the pure audio signals corresponding to multiple audio sources.

[0074] S502: Perform a Fourier transform on the mixed audio signal to obtain the frequency domain signal corresponding to the mixed audio signal.

[0075] The following is combined Figure 6 Let's illustrate with examples. Figure 6 This is a schematic diagram illustrating another audio signal separation process provided in an embodiment of this disclosure. For example... Figure 6As shown, in this embodiment, the mixed audio signal to be processed is assumed to be x, and the pure audio signal (i.e., the first audio signal) corresponding to the first sound source in the mixed audio signal is denoted as s. A short-time Fourier transform is performed on the mixed audio signal x to obtain the frequency domain signal X corresponding to the mixed audio signal x. The frequency domain signal corresponding to the first audio signal s is denoted as S. For ease of distinction, the first audio signal separated using the method of this embodiment will be denoted as... Its corresponding frequency domain signal is denoted as

[0076] S503: Determine the first amplitude information and the first phase information of the mixed audio signal based on the frequency domain signal corresponding to the mixed audio signal.

[0077] See Figure 6 From the frequency domain signal X corresponding to the mixed audio signal, we can obtain the first amplitude information |X| of the mixed audio signal and the first phase information ∠X of the mixed audio signal.

[0078] S504: The first amplitude information is processed by the sound source separation model corresponding to the first sound source to obtain the amplitude difference information and phase difference information between the mixed audio signal and the first audio signal, wherein the first audio signal is the pure audio signal corresponding to the first sound source in the mixed audio signal.

[0079] S505: Determine the second amplitude information based on the first amplitude information and the amplitude difference information.

[0080] The amplitude difference information includes: amplitude ratio coefficient and amplitude residual information.

[0081] In one possible implementation, the product of the first amplitude information and the amplitude scaling factor is determined as the third amplitude information, and the sum of the third amplitude information and the amplitude residual information is determined as the fourth amplitude information; the fourth amplitude information is then subjected to nonlinear processing using a preset activation function to obtain the second amplitude information.

[0082] In this embodiment, the determined second amplitude information can be regarded as the amplitude information of the first audio signal. The following example illustrates the process of determining the second amplitude information.

[0083] In this embodiment, an audio separation method based on a Complex Ideal Ratio Mask (cIRM) is used. That is, cIRM is used to characterize the difference information between the mixed audio signal and the first audio signal. This difference information may include amplitude difference information and phase difference information.

[0084] The following uses To represent cIRM, then It can be expressed using the following formula.

[0085]

[0086] Among them, X r and X i S represents the real and imaginary parts of X, respectively. r and S i Let S be the real part and the imaginary part, respectively. According to the above formula, we can obtain:

[0087]

[0088] The above formula shows that S can be obtained by varying the amplitude and phase of X. Wherein, Indicates the amplitude scaling factor. This indicates phase difference information. The value range of is [0, 1]. Thus, in this embodiment, according to... The first audio signal can be determined by |X| and ∠X.

[0089] However, in the process of realizing this disclosure, the applicant also discovered that by utilizing the above... and The first audio signal obtained after separation still exhibits a certain degree of distortion. The following is a detailed analysis of the causes of this distortion.

[0090] Figure 7A and Figure 7B This is a schematic diagram of the additive noise model provided in an embodiment of this disclosure. For ease of understanding, this embodiment adopts an additive noise model: X(t,f) = S(t,f) + N(t,f). Wherein, X(t,f) represents the mixed audio signal, S(t,f) represents the first audio signal, and N(t,f) represents noise (which can be regarded as the remaining part of the mixed audio signal other than the first audio signal). Figure 7A The amplitude of cIRM is shown. The case where it is less than 1. When When the value range of is [0, 1], it can simulate Figure 7A As shown. However, see... Figure 7B In practical applications, the amplitude of cIRM may also exist. In cases where the value is greater than 1, for example, when the phases of S and N are opposite, the amplitude of the mixed audio signal X will be less than the amplitude of the first audio signal S.

[0091] Therefore, The value range of is [0, 1], which may lead to poor separation results in some scenarios. To further verify the above analysis results, the applicant conducted several sets of comparative experiments.

[0092] In this embodiment, the signal-to-distortion ratio (SDR) is defined using the following formula, and the SDR is used to evaluate the separation performance of the first audio signal. The higher the SDR, the better the separation effect of the first audio signal; the lower the SDR, the worse the separation effect of the first audio signal.

[0093] Ideally, perfect separation would make the SDR infinitely large.

[0094]

[0095] In the above formula, s represents the actual first audio signal. This represents the first audio signal obtained after separation.

[0096] This embodiment uses the MUSDB18 dataset to conduct experiments on multiple sound sources (voice, accompaniment, bass, drums, and other instruments), and the experimental results are shown in Table 1.

[0097] Table 1

[0098]

[0099] The IRM examples in Table 1 illustrate the separation performance of the aforementioned related techniques (i.e., predicting only amplitude difference information and not phase difference information). Here, IRM(1) represents the amplitude scaling factor. The upper limit of the value is 1, and IRM(inf) represents the amplitude scaling factor. There is no upper limit to the possible values.

[0100] Table 1 shows examples of cIRM, illustrating the separation performance using cIRM (i.e., predicting amplitude difference information and phase difference information separately). Here, cIRM(1) represents the amplitude scaling factor. The upper limit of the value is 1, and cIRM(2) represents the amplitude scaling factor. The upper limit of the value is 2, and cIRM(5) represents the amplitude scaling factor. The upper limit of the value is 5, and cIRM(10) represents the amplitude scaling factor. The upper limit of the value is 10, and cIRM(inf) represents the amplitude scaling factor. There is no upper limit to the possible values.

[0101] As can be seen from Table 1, when using IRM, even the amplitude scaling factor... The value of has no upper limit and does not significantly improve separation performance. In this embodiment, using cIRM can significantly improve separation performance, and, with the amplitude scaling factor... Increasing the upper limit of the value significantly improves the separation performance.

[0102] Figure 8 This is a schematic diagram of the distribution of cIRM provided in an embodiment of this disclosure. For example... Figure 8 As shown, the distribution of cIRM for different sound sources (voice, accompaniment, bass, drums, other instruments, and all sound sources) is illustrated. The horizontal axis represents the real part, and the vertical axis represents the imaginary part. Figure 8 Each point in the diagram represents a cIRM. Figure 8 The circle in the image corresponds to a mask with an amplitude of 1. (From...) Figure 8 It can be seen that for each sound source, there are many cIRMs with amplitudes greater than 1. The percentages of cIRM amplitudes greater than 1 for vocals, accompaniment, bass, drums, and other instruments are 20.3%, 34.5%, 6.1%, 26.9%, and 13.9%, respectively.

[0103] In this embodiment, in order to solve the problem caused by the amplitude scaling factor The upper limit of the value of the value leads to poor separation results. In addition to predicting the amplitude ratio coefficient, the sound source separation model also has other limitations. It can also predict amplitude residual information. In other words, the amplitude difference information in this embodiment may include: amplitude scaling factor. and amplitude residual information

[0104] See also Figure 6 The first amplitude information |X| of the mixed audio signal is input into the source separation model corresponding to the first audio source. The source separation model processes the first amplitude information |X| to obtain amplitude difference information and phase difference information. The amplitude difference information includes: amplitude scaling factor. and amplitude residual information According to the amplitude ratio coefficient and amplitude residual information The second amplitude information can be determined using the following formula.

[0105]

[0106] Here, `relu()` is a preset activation function used for nonlinear processing. By using `relu()` for nonlinear processing, the determined second amplitude information is preserved. Greater than 0.

[0107] In this embodiment, the second amplitude information This refers to the amplitude information of the first audio signal. Since the amplitude scaling factor was predicted during the aforementioned process... It also predicted the amplitude residual information. This makes the amplitude information of the determined first audio signal more accurate. Additionally, by predicting the amplitude residual information... This is equivalent to eliminating the amplitude scaling factor. The upper limit of the value solves the problem of amplitude ratio coefficient. The upper limit of the possible values ​​leads to poor separation results.

[0108] S506: Determine the second phase information based on the first phase information and the phase difference information.

[0109] See also Figure 6 The sound source separation model processes the first amplitude information |X| to obtain phase difference information, including: real part of phase information. and phase imaginary part information Thus, based on the phase real part information and phase imaginary part information Phase difference information can be determined.

[0110] Thus, based on the first phase information ∠X and the phase difference information The second phase information can be determined. as follows:

[0111]

[0112] S507: Use the second amplitude information as the amplitude information of the first audio signal and the second phase information as the phase information of the first audio signal to obtain the frequency domain signal corresponding to the first audio signal.

[0113] Specifically, the frequency domain signal corresponding to the first audio signal can be obtained using the following formula.

[0114]

[0115] S508: Perform an inverse Fourier transform on the frequency domain signal corresponding to the first audio signal to obtain the first audio signal.

[0116] See also Figure 6 By analyzing the frequency domain signal corresponding to the first audio signal Perform an inverse Fourier transform to obtain the first audio signal.

[0117] In this embodiment, by predicting the amplitude residual information, the upper limit of the amplitude proportional coefficient is eliminated, thus solving the problem of poor separation effect caused by the upper limit of the amplitude proportional coefficient.

[0118] Based on any of the above embodiments, the structure and training process of the sound source separation model will be introduced below with reference to a specific embodiment.

[0119] Figure 9A This is a schematic diagram of the structure of a sound source separation model provided in an embodiment of the present disclosure. Figure 9B for Figure 9A A schematic diagram of the REB structure in the diagram. Figure 9C for Figure 9A A schematic diagram of the structure of RDB in the diagram. Figure 9D for Figure 9B and Figure 9C A schematic diagram of the RCB structure.

[0120] like Figure 9A As shown, the sound source separation model includes an encoding layer, an intermediate layer, and a decoding layer. The encoding and intermediate layers are used to extract features from the first amplitude information, and the decoding layer is used to make predictions based on the feature extraction results to obtain amplitude difference information and phase difference information.

[0121] See 9A. Figure 9B and Figure 9D The coding layer consists of K residual encoder blocks (REBs), where K is an integer greater than 1. Figure 9A The example uses K=6. Each REB consists of four consecutively connected Residual Convolutional Blocks (RCBs) and a pooling layer (Avg pool). Each RCB includes two convolutional layers (Conv) with a kernel size of 3*3. Before each convolutional layer, there is a normalization layer (BN) and an activation function layer (Leaky_relu). A convolutional layer (Conv) is added between the input and output of the RCB. Each REB also includes a 2*2 pooling layer after the four RCBs to reduce the number of features. Therefore, each REB consists of eight convolutional layers.

[0122] See Figure 9A , Figure 9C and Figure 9D The decoding layer consists of K residual decoder blocks (RDBs). Figure 9A The example shown uses K=6. Each RDB is symmetrical to the REB. Each RDB consists of a transposed convolutional layer and four RCBs connected in sequence. The transposed convolutional layer has a kernel size of 3*3 and is used for upsampling features. The RCBs in the RDB have the same structure as those in the REB. Thus, each RDB contains nine convolutional layers.

[0123] In some possible implementations, to further improve the model's feature representation capabilities, an intermediate layer can be introduced between the encoding and decoding layers. For example... Figure 9A As shown, the intermediate layer consists of T intermediate convolutional blocks (ICBs), where T is an integer greater than 1. Each ICB contains 4 RCBs. Thus, each ICB contains 8 convolutional layers.

[0124] Furthermore, a normalization layer (BN) can be included before the encoding layer. After the decoding layer, one ICB and an output convolutional layer with J output channels can be included. The value of J is related to the number of output parameters of the audio source separation model. In this embodiment, when four parameters need to be output (e.g., ...), ... Figure 6 In When ), set J to 8.

[0125] It should be understood that, Figure 9A In this context, the numbers labeled in each layer or block represent the number of features. For example, 1025 in a BN layer indicates that there are 1025 features.

[0126] In this embodiment, network depth is increased by using multiple REBs and multiple RDBs. See also Figure 9A When K=6 and T=4, the audio source separation model has a total of 143 convolutional layers, which can greatly improve the audio separation effect.

[0127] Combination Figure 9A The structure of the sound source separation model is shown below. The processing procedure of the sound source separation model will be explained below.

[0128] In one example, such as Figure 9A As shown, when the sound source separation model does not include an intermediate layer, the first amplitude information is processed by the encoding layer to obtain a first intermediate result; the first intermediate result is processed by the decoding layer to obtain the amplitude difference information and phase difference information.

[0129] In another example, such as Figure 9A As shown, when the sound source separation model includes an intermediate layer, the first amplitude information is processed by the encoding layer to obtain the first intermediate result; the first amplitude information is processed by the encoding layer and the intermediate layer to obtain the second intermediate result; the first intermediate result and the second intermediate result are processed by the decoding layer to obtain the amplitude difference information and the phase difference information.

[0130] The following is about Figure 9A The training process of the sound source separation model shown is explained.

[0131] In this embodiment, the MUSDB18 dataset is used. Figure 9A The audio source separation model shown was tested. The MUSDB18 dataset includes clean audio signals for individual vocals, accompaniment, bass, drums, and other instruments. These clean audio signals were divided into 3-second segments. By randomly mixing these segments from different audio sources, a mixed audio signal x was obtained. A short-time Fourier transform was performed on the mixed audio signal x to obtain the frequency domain signal. This frequency domain signal was then input into the audio separation model for training.

[0132] It should be noted that, since the audio source separation model in this embodiment adopts a network structure based on convolutional layers, it does not require previous states to calculate the current prediction. Therefore, the audio source separation model supports parallel processing of multiple mixed audio signals.

[0133] For example, during training, the batch size is set to 16, and the Adaptive Moment Estimation (Adam) optimizer is applied. The learning rates are set to 0:001, 0:0005, 0:0001, 0:0002, and 0:0005 for vocals, accompaniment, bass, drums, and other instruments, respectively. These learning rates are adjusted based on the validation set of the MUSDB18 dataset. The learning rate is multiplied by a factor of 0.9 every 15,000 steps. After 300,000 steps of training, the trained sound source separation model is obtained.

[0134] Figure 10 This is a schematic diagram of an audio signal separation device provided in an embodiment of this disclosure. The device can be in the form of software and / or hardware. For example... Figure 10 As shown, the audio signal separation device 1000 provided in this embodiment includes: a first determining module 1001, a processing module 1002, and a second determining module 1003.

[0135] The first determining module 1001 is used to determine the first amplitude information of the mixed audio signal to be processed and the first phase information of the mixed audio signal, wherein the mixed audio signal is formed by mixing pure audio signals corresponding to multiple sound sources;

[0136] Processing module 1002 is used to process the first amplitude information to obtain amplitude difference information and phase difference information between the mixed audio signal and the first audio signal, wherein the first audio signal is the pure audio signal corresponding to the first sound source in the mixed audio signal;

[0137] The second determining module 1003 is used to determine the first audio signal based on the first amplitude information, the first phase information, the amplitude difference information, and the phase difference information.

[0138] In one possible implementation, the second determining module 1003 is specifically used for:

[0139] Based on the first amplitude information and the amplitude difference information, the second amplitude information is determined;

[0140] The second phase information is determined based on the first phase information and the phase difference information;

[0141] The first audio signal is determined based on the second amplitude information and the second phase information.

[0142] In one possible implementation, the amplitude difference information includes: amplitude scaling factor and amplitude residual information; the second determining module 1003 is specifically used for:

[0143] The product of the first amplitude information and the amplitude ratio coefficient is determined as the third amplitude information;

[0144] The sum of the third amplitude information and the amplitude residual information is determined as the fourth amplitude information;

[0145] The fourth amplitude information is processed nonlinearly using a preset activation function to obtain the second amplitude information.

[0146] In one possible implementation, the second determining module 1003 is specifically used for:

[0147] The second amplitude information is used as the amplitude information of the first audio signal, and the second phase information is used as the phase information of the first audio signal to obtain the first frequency domain signal corresponding to the first audio signal.

[0148] The first audio signal is obtained by performing an inverse Fourier transform on the first frequency domain signal.

[0149] In one possible implementation, the first determining module 1001 is specifically used for:

[0150] Perform a Fourier transform on the mixed audio signal to obtain the second frequency domain signal corresponding to the mixed audio signal;

[0151] The first amplitude information and the first phase information are determined based on the second frequency domain signal.

[0152] In one possible implementation, the processing module 1002 is specifically used to: process the first amplitude information through the sound source separation model corresponding to the first sound source to obtain amplitude difference information and phase difference information between the mixed audio signal and the first audio signal;

[0153] The sound source separation model is obtained by training multiple sets of training samples. Each set of training samples includes: amplitude information of the sample mixed audio signal, amplitude difference information and phase difference information between the sample mixed audio signal and the sample pure audio signal, and the sample pure audio signal is the pure audio signal corresponding to the first sound source in the sample mixed audio signal.

[0154] In one possible implementation, the audio source separation model includes an encoding layer and a decoding layer, the encoding layer includes multiple residual coded blocks (REBs), and the decoding layer includes multiple residual decoded blocks (RDBs); the processing module 1002 is specifically used for:

[0155] The first amplitude information is processed by the encoding layer to obtain a first intermediate result;

[0156] The first intermediate result is processed by the decoding layer to obtain the amplitude difference information and the phase difference information.

[0157] In one possible implementation, the audio source separation model further includes an intermediate layer, which is disposed between the encoding layer and the decoding layer, and the intermediate layer includes multiple intermediate convolutional blocks (ICBs); the processing module is specifically used for:

[0158] The first amplitude information is processed through the encoding layer and the intermediate layer to obtain a second intermediate result;

[0159] The first intermediate result and the second intermediate result are processed by the decoding layer to obtain the amplitude difference information and the phase difference information.

[0160] The audio signal separation device provided in this embodiment can be used to execute the audio signal separation method provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0161] To implement the above embodiments, this disclosure also provides an electronic device.

[0162] refer to Figure 11The diagram illustrates a structural schematic of an electronic device 1100 suitable for implementing embodiments of the present disclosure. The electronic device 1100 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 11 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0163] like Figure 11 As shown, electronic device 1100 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1101, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1102 or a program loaded from storage device 1108 into random access memory (RAM) 1103. RAM 1103 also stores various programs and data required for the operation of electronic device 1100. The processing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0164] Typically, the following devices can be connected to I / O interface 1105: input devices 1106 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1107 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1108 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1109. Communication device 1109 allows electronic device 1100 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 11 An electronic device 1100 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0165] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1109, or installed from storage device 1108, or installed from ROM 1102. When the computer program is executed by processing device 1101, it performs the functions defined in the methods of embodiments of this disclosure.

[0166] It should be noted that the computer-readable storage medium described in this disclosure can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0167] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0168] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0169] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0170] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0171] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0172] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0173] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0174] In a first aspect, according to one or more embodiments of the present disclosure, a method for separating audio signals is provided, comprising:

[0175] The first amplitude information and the first phase information of the mixed audio signal to be processed are determined, wherein the mixed audio signal is formed by mixing pure audio signals corresponding to multiple sound sources;

[0176] The first amplitude information is processed to obtain amplitude difference information and phase difference information between the mixed audio signal and the first audio signal, wherein the first audio signal is the pure audio signal corresponding to the first sound source in the mixed audio signal;

[0177] The first audio signal is determined based on the first amplitude information, the first phase information, the amplitude difference information, and the phase difference information.

[0178] According to one or more embodiments of this disclosure, determining the first audio signal based on the first amplitude information, the first phase information, the amplitude difference information, and the phase difference information includes:

[0179] Based on the first amplitude information and the amplitude difference information, the second amplitude information is determined;

[0180] The second phase information is determined based on the first phase information and the phase difference information;

[0181] The first audio signal is determined based on the second amplitude information and the second phase information.

[0182] According to one or more embodiments of this disclosure, the amplitude difference information includes: amplitude scaling factor and amplitude residual information; determining the second amplitude information based on the first amplitude information and the amplitude difference information includes:

[0183] The product of the first amplitude information and the amplitude ratio coefficient is determined as the third amplitude information;

[0184] The sum of the third amplitude information and the amplitude residual information is determined as the fourth amplitude information;

[0185] The fourth amplitude information is processed nonlinearly using a preset activation function to obtain the second amplitude information.

[0186] According to one or more embodiments of this disclosure, determining the first audio signal based on the second amplitude information and the second phase information includes:

[0187] The second amplitude information is used as the amplitude information of the first audio signal, and the second phase information is used as the phase information of the first audio signal to obtain the first frequency domain signal corresponding to the first audio signal.

[0188] The first audio signal is obtained by performing an inverse Fourier transform on the first frequency domain signal.

[0189] According to one or more embodiments of this disclosure, determining the first amplitude information of the mixed audio signal to be processed and the first phase information of the mixed audio signal includes:

[0190] Perform a Fourier transform on the mixed audio signal to obtain the second frequency domain signal corresponding to the mixed audio signal;

[0191] The first amplitude information and the first phase information are determined based on the second frequency domain signal.

[0192] According to one or more embodiments of this disclosure, processing the first amplitude information to obtain amplitude difference information and phase difference information between the mixed audio signal and the first audio signal includes:

[0193] The amplitude information is processed by the sound source separation model corresponding to the first sound source to obtain the amplitude difference information and phase difference information between the mixed audio signal and the first audio signal;

[0194] The sound source separation model is obtained by training multiple sets of training samples. Each set of training samples includes: amplitude information of the sample mixed audio signal, amplitude difference information and phase difference information between the sample mixed audio signal and the sample pure audio signal, and the sample pure audio signal is the pure audio signal corresponding to the first sound source in the sample mixed audio signal.

[0195] According to one or more embodiments of this disclosure, the audio source separation model includes an encoding layer and a decoding layer, the encoding layer including a plurality of residual coded blocks (REBs) and the decoding layer including a plurality of residual decoded blocks (RDBs);

[0196] The step of processing the first amplitude information using the sound source separation model corresponding to the first sound source to obtain amplitude difference information and phase difference information between the mixed audio signal and the first audio signal includes:

[0197] The first amplitude information is processed by the encoding layer to obtain a first intermediate result;

[0198] The first intermediate result is processed by the decoding layer to obtain the amplitude difference information and the phase difference information.

[0199] According to one or more embodiments of this disclosure, the audio source separation model further includes an intermediate layer disposed between the encoding layer and the decoding layer, the intermediate layer including a plurality of intermediate convolutional blocks (ICBs);

[0200] The step of processing the first intermediate result through the decoding layer to obtain the amplitude difference information and phase difference information includes:

[0201] The first amplitude information is processed through the encoding layer and the intermediate layer to obtain a second intermediate result;

[0202] The first intermediate result and the second intermediate result are processed by the decoding layer to obtain the amplitude difference information and the phase difference information.

[0203] Secondly, according to one or more embodiments of this disclosure, an audio signal separation device is provided, comprising:

[0204] The first determining module is used to determine the first amplitude information of the mixed audio signal to be processed and the first phase information of the mixed audio signal, wherein the mixed audio signal is formed by mixing pure audio signals corresponding to multiple sound sources;

[0205] The processing module is used to process the first amplitude information to obtain amplitude difference information and phase difference information between the mixed audio signal and the first audio signal, wherein the first audio signal is the pure audio signal corresponding to the first sound source in the mixed audio signal;

[0206] The second determining module is used to determine the first audio signal based on the first amplitude information, the first phase information, the amplitude difference information, and the phase difference information.

[0207] According to one or more embodiments of this disclosure, the second determining module is specifically used for:

[0208] Based on the first amplitude information and the amplitude difference information, the second amplitude information is determined;

[0209] The second phase information is determined based on the first phase information and the phase difference information;

[0210] The first audio signal is determined based on the second amplitude information and the second phase information.

[0211] According to one or more embodiments of this disclosure, the amplitude difference information includes: amplitude scaling factor and amplitude residual information; the second determining module is specifically used for:

[0212] The product of the first amplitude information and the amplitude ratio coefficient is determined as the third amplitude information;

[0213] The sum of the third amplitude information and the amplitude residual information is determined as the fourth amplitude information;

[0214] The fourth amplitude information is processed nonlinearly using a preset activation function to obtain the second amplitude information.

[0215] According to one or more embodiments of this disclosure, the second determining module is specifically used for:

[0216] The second amplitude information is used as the amplitude information of the first audio signal, and the second phase information is used as the phase information of the first audio signal to obtain the first frequency domain signal corresponding to the first audio signal.

[0217] The first audio signal is obtained by performing an inverse Fourier transform on the first frequency domain signal.

[0218] According to one or more embodiments of this disclosure, the first determining module is specifically used for:

[0219] Perform a Fourier transform on the mixed audio signal to obtain the second frequency domain signal corresponding to the mixed audio signal;

[0220] The first amplitude information and the first phase information are determined based on the second frequency domain signal.

[0221] According to one or more embodiments of this disclosure, the processing module is specifically used for:

[0222] The amplitude information is processed by the sound source separation model corresponding to the first sound source to obtain the amplitude difference information and phase difference information between the mixed audio signal and the first audio signal;

[0223] The sound source separation model is obtained by training multiple sets of training samples. Each set of training samples includes: amplitude information of the sample mixed audio signal, amplitude difference information and phase difference information between the sample mixed audio signal and the sample pure audio signal, and the sample pure audio signal is the pure audio signal corresponding to the first sound source in the sample mixed audio signal.

[0224] According to one or more embodiments of this disclosure, the audio source separation model includes an encoding layer and a decoding layer, the encoding layer includes multiple residual coded blocks (REBs), and the decoding layer includes multiple residual decoded blocks (RDBs); the processing module is specifically used for:

[0225] The first amplitude information is processed by the encoding layer to obtain a first intermediate result;

[0226] The first intermediate result is processed by the decoding layer to obtain the amplitude difference information and the phase difference information.

[0227] According to one or more embodiments of this disclosure, the audio source separation model further includes an intermediate layer disposed between the encoding layer and the decoding layer, the intermediate layer including a plurality of intermediate convolutional blocks (ICBs); the processing module is specifically used for:

[0228] The first amplitude information is processed through the encoding layer and the intermediate layer to obtain a second intermediate result;

[0229] The first intermediate result and the second intermediate result are processed by the decoding layer to obtain the amplitude difference information and the phase difference information.

[0230] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0231] The memory stores computer-executed instructions;

[0232] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method described in the first aspect above and various possible designs of the first aspect.

[0233] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, which, when executed by a processor, implement the method described in the first aspect above and various possible designs of the first aspect.

[0234] Fifthly, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described in the first aspect and various possible designs of the first aspect.

[0235] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0236] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0237] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for separating audio signals, characterized in that, include: The first amplitude information and the first phase information of the mixed audio signal to be processed are determined, wherein the mixed audio signal is formed by mixing pure audio signals corresponding to multiple sound sources; The first amplitude information is processed to obtain amplitude difference information and phase difference information between the mixed audio signal and the first audio signal, wherein the first audio signal is the pure audio signal corresponding to the first sound source in the mixed audio signal; Based on the first amplitude information and the amplitude difference information, the second amplitude information is determined; The second phase information is determined based on the first phase information and the phase difference information; The first audio signal is determined based on the second amplitude information and the second phase information; The amplitude difference information includes: amplitude scaling factor and amplitude residual information; determining the second amplitude information based on the first amplitude information and the amplitude difference information includes: The product of the first amplitude information and the amplitude ratio coefficient is determined as the third amplitude information; The sum of the third amplitude information and the amplitude residual information is determined as the fourth amplitude information; The fourth amplitude information is processed nonlinearly using a preset activation function to obtain the second amplitude information.

2. The method according to claim 1, characterized in that, Determining the first audio signal based on the second amplitude information and the second phase information includes: The second amplitude information is used as the amplitude information of the first audio signal, and the second phase information is used as the phase information of the first audio signal to obtain the first frequency domain signal corresponding to the first audio signal. The first audio signal is obtained by performing an inverse Fourier transform on the first frequency domain signal.

3. The method according to claim 1 or 2, characterized in that, The determination of the first amplitude information of the mixed audio signal to be processed and the first phase information of the mixed audio signal include: Perform a Fourier transform on the mixed audio signal to obtain the second frequency domain signal corresponding to the mixed audio signal; The first amplitude information and the first phase information are determined based on the second frequency domain signal.

4. The method according to claim 1 or 2, characterized in that, The step of processing the first amplitude information to obtain amplitude difference information and phase difference information between the mixed audio signal and the first audio signal includes: The amplitude information is processed by the sound source separation model corresponding to the first sound source to obtain the amplitude difference information and phase difference information between the mixed audio signal and the first audio signal; The sound source separation model is obtained by training multiple sets of training samples. Each set of training samples includes: amplitude information of the sample mixed audio signal, amplitude difference information and phase difference information between the sample mixed audio signal and the sample pure audio signal, and the sample pure audio signal is the pure audio signal corresponding to the first sound source in the sample mixed audio signal.

5. The method according to claim 4, characterized in that, The audio source separation model includes an encoding layer and a decoding layer. The encoding layer includes multiple residual coded blocks (REBs), and the decoding layer includes multiple residual decoded blocks (RDBs). The step of processing the first amplitude information using the sound source separation model corresponding to the first sound source to obtain amplitude difference information and phase difference information between the mixed audio signal and the first audio signal includes: The first amplitude information is processed by the encoding layer to obtain a first intermediate result; The first intermediate result is processed by the decoding layer to obtain the amplitude difference information and the phase difference information.

6. The method according to claim 5, characterized in that, The audio source separation model also includes an intermediate layer, which is disposed between the encoding layer and the decoding layer, and the intermediate layer includes multiple intermediate convolutional blocks (ICBs). The step of processing the first intermediate result through the decoding layer to obtain the amplitude difference information and phase difference information includes: The first amplitude information is processed through the encoding layer and the intermediate layer to obtain a second intermediate result; The first intermediate result and the second intermediate result are processed by the decoding layer to obtain the amplitude difference information and the phase difference information.

7. An audio signal separation device, characterized in that, include: The first determining module is used to determine the first amplitude information of the mixed audio signal to be processed and the first phase information of the mixed audio signal, wherein the mixed audio signal is formed by mixing pure audio signals corresponding to multiple sound sources; The processing module is used to process the first amplitude information to obtain amplitude difference information and phase difference information between the mixed audio signal and the first audio signal, wherein the first audio signal is the pure audio signal corresponding to the first sound source in the mixed audio signal; The second determining module is used for Based on the first amplitude information and the amplitude difference information, determine the second amplitude information; based on the first phase information and the phase difference information, determine the second phase information; The first audio signal is determined based on the second amplitude information and the second phase information; The amplitude difference information includes: amplitude scaling factor and amplitude residual information; the second determining module is specifically used for: The product of the first amplitude information and the amplitude ratio coefficient is determined as the third amplitude information; The sum of the third amplitude information and the amplitude residual information is determined as the fourth amplitude information; The fourth amplitude information is processed nonlinearly using a preset activation function to obtain the second amplitude information.

8. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes the computer execution instructions to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 6.

Images