Signal extraction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a signal extraction method and device, electronic equipment and a storage medium, and belongs to the technical field of signal processing. The method comprises the following steps: inputting a mixed signal to be separated into a separator model, and extracting a first extracted signal; wherein the separator model is obtained based on the following training steps: inputting the first mixed signal into the separator model to obtain a first output signal; inputting the first output signal and the target signal into a discriminator model for discrimination, and calculating the loss of the current separator model and the loss of the current discriminator model; and performing alternate confrontation training on the separator model and the discriminator model based on the current separator model loss and the current discriminator model loss. According to the method and the device, the separation performance of the separator model can be improved, and then relatively pure human voice signals are extracted.

Description

Technical Field

[0001] The present application relates to the field of signal processing technology, and in particular to a signal extraction method, device, electronic device and storage medium. Background Art

[0002] In audio usage scenarios such as vocal enhancement, karaoke, and stereo to 5.1 surround sound, the audio quality can be improved and the desired sound effects can be achieved by extracting the vocals and accompaniment from the mixed music signal and then processing the extracted vocals and accompaniment separately as required. It can be seen that separating the vocals and accompaniment from the mixed music signal provides a basis for improving the sound quality and adjusting the sound effects.

[0003] Generally, the human voice and accompaniment in a mixed music signal can be separated based on non-negative matrix factorization technology. With the development of deep learning, the task of separating human voice and accompaniment in a mixed music signal can also be achieved based on deep neural networks.

[0004] However, the existing deep neural network does not separate the human voice and accompaniment in the mixed music signal cleanly. In other words, the existing human voice accompaniment separation method has a poor separation effect and cannot extract a relatively pure human voice or a relatively pure accompaniment. Summary of the invention

[0005] The embodiments of the present application provide a signal extraction method, device, electronic device and storage medium, which can solve the problem of poor separation of vocals and accompaniment and inability to extract pure vocals or accompaniment. To achieve the above purpose, the technical solutions provided by the embodiments of the present application are as follows:

[0006] In a first aspect, an embodiment of the present application provides a signal extraction method, the signal extraction method comprising:

[0007] The mixed signal to be separated is input into a separator model to extract a first extracted signal; wherein the separator model is obtained based on the following training steps:

[0008] Inputting a first mixed signal into the separator model to obtain a first output signal;

[0009] Input the first output signal and the target signal into the discriminator model for discrimination, and calculate the current separator model loss and the current discriminator model loss;

[0010] Based on the current separator model loss and the current discriminator model loss, the separator model and the discriminator model are alternately trained against each other.

[0011] In a second aspect, an embodiment of the present application provides a signal extraction device, the signal extraction device comprising:

[0012] A separation module, used to input the mixed signal to be separated into a separator model to extract a first extracted signal; and also used to input the first mixed signal into the separator model to obtain a first output signal;

[0013] A loss calculation module, used for inputting the first output signal and the target signal into the discriminator model for discrimination, and calculating the current separator model loss and the current discriminator model loss;

[0014] An adversarial training module is used to perform alternating adversarial training on the separator model and the discriminator model based on the current separator model loss and the current discriminator model loss.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the signal extraction method as described in the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the signal extraction method as described in the first aspect.

[0017] In the embodiment of the present application, the separator model can be combined with the discriminator model, and the loss function is used to perform adversarial training on the separator model and the discriminator model, so that the separator model and the discriminator model can promote each other, thereby extracting an ideal output signal based on the separator model that completes the training. It can be understood that when the present application is used to separate the mixed music signal from the human voice and the accompaniment, based on the alternating adversarial training of the separator model and the discriminator model, the separator model and the discriminator model promote and game each other, and the separator model after completing the adversarial training can output a signal close to the target signal (such as a pure human voice signal) as much as possible, so that the discriminator model is difficult to judge the signal output by the separator model as false. Therefore, after the mixed music signal is input in the separator model that completes the adversarial training, a relatively pure human voice signal can be output, and the separation effect of the signal extraction model for the human voice and the accompaniment is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram of a flow chart of a signal extraction method provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a signal processing flow in a signal extraction method provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of a separator model structure provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of the structure of a residual block in a separator model provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of the structure of a signal extraction device provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0026] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0027] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0028] In the description of the present application, it should be understood that the terms "first" and "second" etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0029] At present, the mixed signal separation method based on deep neural network has the problem of poor separation effect. For example, when separating the vocals and accompaniment in a song, there is a situation where the vocals and accompaniment are not separated cleanly. In particular, when there is only accompaniment in a song, after the vocal accompaniment separation process of the song, the audio amplitude (i.e. the amplitude of the sound wave) of the obtained vocal signal is not zero, and there will be some leakage of the accompaniment. In other words, when the existing deep neural network is used to separate the vocal accompaniment of a song, the separated vocals are still mixed with part of the accompaniment signal, or the separated accompaniment is still mixed with part of the vocal signal.

[0030] Based on this, the present application provides a signal extraction method, device, electronic device and storage medium. By combining the separator model with the discriminator model and using the loss function to perform adversarial training on the separator model and the discriminator model, the separator model and the discriminator model can promote each other, thereby extracting an ideal output signal based on the trained separator model, thereby improving the signal separation effect of the separator model.

[0031] It should be noted that the signal extraction method provided in the present application can be applied to any scenario of separating multiple signals from a mixed signal. For example, the above-mentioned signal separation scenario can be: extracting vocal signals and / or accompaniment signals from songs, extracting individual voice signals from mixed voices of multiple people, extracting clean voice signals from voice signals containing noise, etc. The following will take the extraction of vocal signals and accompaniment signals from songs as an example to explain the present application in detail.

[0032] In implementation, the signal extraction method may include inputting a mixed signal into a separator model, and extracting a desired type of signal through the separator model. The separator model is obtained based on a training step. For ease of description, the actual mixed signal to be separated processed by the separator model in the inference stage may be referred to as the mixed signal to be separated, and the signal output by the separator model for the mixed signal to be separated may be referred to as the first extracted signal. In addition, any mixed signal from the training data set processed by the separator model in the training stage may be referred to as the first mixed signal, and the signal output by the separator model for the first mixed signal may be referred to as the first output signal.

[0033] In one embodiment, see Figure 1 , a signal extraction method provided in the present application may include the following steps:

[0034] S100, inputting a first mixed signal into a separator model to be trained to obtain a first output signal.

[0035] In implementation, when the mixed music signal obtained from the training data set is input into the separator model to be trained for the first time, the separator model can randomly generate an output signal. The separator model can be trained several times based on several mixed signals in the training data set to construct a trained separator model. Afterwards, the first extracted signal of the required type can be extracted from the mixed signal to be separated based on the trained separator model. For the convenience of description, the separator model before the training is completed can be referred to as the separator model to be trained.

[0036] In one embodiment, the first mixed signal may be a mixed music signal including a vocal signal and an accompaniment signal, and the first output signal may be a vocal signal.

[0037] In one embodiment, step S100 may specifically include steps S110 and S120 (not shown in the figure).

[0038] S110, performing data preprocessing on the mixed signal and then inputting the mixed signal into a separator model to obtain a mask matrix.

[0039] S120, performing a transformation corresponding to the data preprocessing on the mask matrix to obtain a first output signal.

[0040] In implementation, the mixed signal may be subjected to data conversion processing and then input into the separator model. Accordingly, after the separator model outputs a signal, the output signal may be subjected to corresponding inverse processing to obtain a desired first output signal.

[0041] S200, inputting the first output signal and the target signal into the discriminator model for discrimination, and calculating the current separator model loss and the current discriminator model loss.

[0042] The pre-prepared training data set may include a large number of mixed signals and pure target signals corresponding to each mixed signal. The type of the target signal may determine the type of signal output by the separator model. In other words, the target signal may be prepared in advance based on the type of signal that needs to be extracted in the end. For example, if a pure vocal signal needs to be extracted from a mixed music signal, the pre-prepared target signal in the training data set is a vocal signal.

[0043] In implementation, the first output signal and the target signal corresponding to the first output signal can be input into the discriminator model together to determine whether it is true or false. For example, the first mixed signal can be any mixed music signal obtained from the training database, and the first output signal can be a vocal signal output by the separator model to be trained based on the first mixed signal. The vocal signal output by the separator model needs to be as close to the pure vocal signal as possible. Therefore, the target signal corresponding to the first output signal can be a pure vocal signal obtained from the training database. During the model training stage, there is a certain gap between the first output signal and the target signal. Therefore, after the first output signal and the target signal are input into the discriminator, the discriminator's recognition result for the first output signal can be false, while the recognition result for the target signal can be true. Among them, the discriminator network can use labels to mark the recognition results. For example, the signal recognized as true is marked as 1, and the signal recognized as false is marked as 0.

[0044] It can be understood that there may be a certain degree of inconsistency between the human voice signal output by the separator model and the real pure human voice signal. At this time, the degree of inconsistency between the first output signal of the separator model and the target signal can be estimated by the discrimination result of the discriminator, that is, the loss of the separator model for the current signal extraction process is calculated by the discriminator (which can be called the current separator model loss), and then the current separator model loss is used to train the separator model to improve the signal separation effect of the separator model. Of course, the discriminator can also be used to calculate the loss of the discriminator model for the current signal true and false discrimination process (which can be called the current discriminator model loss), and the current discriminator model loss is used to train the discriminator model to improve the discrimination effect of the discriminator model.

[0045] It is worth mentioning that the purpose of training the separator model is to make the output signal be able to be judged as true by the discriminator model. The purpose of training the discriminator model is to judge the real target signal as true and the signal output by the separator model as false. Therefore, when calculating the loss of the current separator model, the human voice signal output by the current separator model can be marked as true; when calculating the loss of the current discriminator model, the target signal can be marked as true and the human voice signal output by the current separator model can be marked as false.

[0046] S300, based on the current separator model loss and the current discriminator model loss, performing alternating adversarial training on the separator model and the discriminator model.

[0047] In practice, the initial separator model does not have a good separation effect on mixed signals. After the mixed music signal obtained from the training data set is input into the initial separator model, the extracted vocal signal is quite different from the real pure vocal signal, and it is easy to be judged as false by the discriminator model. Therefore, initially, the first output signal output by the separator model and the target signal can be directly input into the discriminator model for training. Afterwards, by alternately training the separator model and the discriminator model, a separator model with better separation effect can be quickly obtained.

[0048] It is worth mentioning that during the alternating adversarial training of the separator model and the discriminator model, if the training data set has sufficient training samples, after updating the model parameters of any model based on one adversarial training, a new mixed signal can be obtained to train the other model next time. If the training data set has fewer training samples, the training data set can be reused to perform alternating adversarial training on the separator model and the discriminator model after all the training samples in the training data set have been used once.

[0049] In one embodiment, step S300 may specifically include: when the current separator model loss is calculated, the model parameters of the discriminator model are fixed, and the current separator model loss is back-propagated to the separator model to adjust the model parameters of the separator model; when the current discriminator model loss is calculated, the model parameters of the separator model are fixed, and the current discriminator model loss is back-propagated to the discriminator model to adjust the model parameters of the discriminator model.

[0050] In implementation, after calculating the loss function value (loss) of the separator model or the discriminator model, the separator model and the discriminator model can back-propagate the loss to update the weight coefficients of the corresponding models. Specifically, back-propagating the loss of the separator model once can update the weight coefficient of the separator model, and then back-propagating the loss of the discriminator model once can update the weight coefficient of the discriminator model. In this way, the separator model and the discriminator model are trained alternately and cyclically until the model converges.

[0051] S400: Input the mixed signal to be separated into the trained separator model to extract a first extracted signal.

[0052] It can be understood that through the above-mentioned adversarial training, the separator model can output a human voice signal that is as close to the pure human voice signal as possible. In this way, after the training of the separator model is completed, when the mixed music signal is input to the separator model, it can output the first extracted signal that is "indistinguishable from the real thing".

[0053] In one embodiment, step S400 may specifically include steps S410 and S420 (not shown in the figure).

[0054] S410, performing data preprocessing on the mixed signal and then inputting the mixed signal into a separator model to obtain a mask matrix.

[0055] S420, performing a transformation corresponding to the data preprocessing on the mask matrix to obtain a first extracted signal.

[0056] In one embodiment, it is possible to determine when the separator model has completed training based on the loss function value of the separator. Accordingly, before step S400, the signal extraction method may further include the following steps: calculating the loss difference between the current separator model loss and the loss of the historical separator model calculated last time; when the loss difference is not higher than a preset difference, determining that the separator model to be trained is the separator model that has completed training.

[0057] In implementation, when the loss of the separator model tends to be stable, that is, when the loss of the separator model basically no longer decreases, the separator model will have a better separation effect. At this time, the separator can be stopped from being trained and used as a trained separator model to extract the final required output signal from the mixed signal to be separated. Of course, it is also possible to determine when the separator model has completed training based on other iterative termination strategies. For example, the training can be terminated when the number of training times of the separator model reaches a preset number of training times, or the training can be terminated when the training duration of the separator reaches a preset training duration, etc. This application does not limit this.

[0058] It is worth mentioning that the separator model and the discriminator model are constantly undergoing alternating adversarial training, which will promote and restrain each other. In this case, the loss of the separator model may first become smaller, then increase, and may fluctuate within a certain range during the adversarial training process. Therefore, the loss of the separator model basically no longer decreases, which can be understood as: the decrease is very small or the loss of the separator model fluctuates within a small range after decreasing to a certain value.

[0059] In one embodiment, in order to obtain other signals (which may be referred to as second extracted signals) in the mixed signal to be separated, after step S400, the first extracted signal may be subtracted from the mixed signal to be separated to extract the second extracted signal.

[0060] See also Figure 2This embodiment provides specific steps for adversarial training of a separator model and a discriminator model. The separator model may be a mask network model, and the discriminator model may be a GAN (Generative adversarial network) model. Steps S111-S114 are specific processing of step S110, steps S121-S123 are specific processing of step S120, and steps S201-S202 are specific processing of step S200.

[0061] S111, input the mixed music signal x into the separator model mix , and for the mixed music signal x mix Perform short-time Fourier transform analysis to obtain the complex spectrum of the mixed signal.

[0062] In one embodiment, the separator model may specifically adopt a U-shaped residual network model. Figure 3 , the separator model may include several encoder blocks, several decoder blocks, and an intermediate layer that serves as a connection, and the intermediate layer may use a convolutional block. For example, Figure 3 The embodiment shown uses 4 Encoder modules and 4 Decoder modules. The Encoder module consists of residual convolution and average pooling layer; the Decoder module consists of transposed convolution and residual convolution; and the intermediate Conv module consists of residual convolution. Figure 3 In the diagram, the symbol “->” indicates the number of channels in each layer of the module. For example, “1->32” means that the number of input channels is 1 and the number of output channels is 32; “32->1” means that the number of input channels is 32 and the number of output channels is 1.

[0063] In this embodiment, the structures of the residual convolution in the Encoder module, the Decoder module, and the intermediate Conv module can be seen in Figure 4 , including the normalization layer Batch Norm, the activation function layer Leaky_relu and the convolution layer Conv2d. In addition, the size of the convolution kernel of the residual convolution is 3×3. In addition, the size of the convolution kernel of the average pooling layer of the Encoder module is 2×2, and the size of the convolution kernel of the transposed layer of the Decoder module is 3×3.

[0064] It should be noted that each module can also adopt other convolution kernel sizes, such as 1x1, 5x5, 7x7, etc., and this application does not impose any restrictions on this.

[0065] S112, obtain the amplitude spectrum X of the mixed signal according to the complex spectrum diagram mix and the original phase

[0066] In practice, the complex spectrum of a signal (complex spectrum for short) includes two parts: amplitude spectrum and phase spectrum. The amplitude spectrum can be understood as the relationship curve between the amplitude and frequency obtained by taking the amplitude of the complex spectrum, and the phase spectrum can be understood as the relationship curve between the phase and frequency obtained by taking the phase of the complex spectrum.

[0067] S113, the amplitude spectrum X mix Enter the separator in the separator model.

[0068] S114, the separator outputs a mask matrix M.

[0069] S121, use the mask matrix M and the amplitude spectrum X of the original signal mix The output amplitude spectrum is obtained by point multiplication, that is, X1 = X mix ·M

[0070] S122, using the output amplitude spectrum X1 and the original phase The composed complex spectrum is used as the output complex spectrum of the separator.

[0071] S123, performing short-time inverse Fourier transform on the output complex spectrum to obtain a first output signal x1 of the separator.

[0072] S201, input the output signal x1 of the separator and the target signal x0 into the discriminator.

[0073] In one embodiment, the discriminator model may specifically adopt the HIFIGAN discriminator model. The HIFIGAN discriminator model may include two discriminators, one is a multi-period discriminator (MPD), which can be used to identify signals of different periods, and the other is a multi-scale discriminator (MSD), which can be used to deal with ultra-long data.

[0074] In implementation, a multi-period discriminator may include several sub-discriminators, each with a different period p. The sub-discriminator first converts the audio into 2-D data according to the value of the period p, and then uses a CNN (Convolutional Neural Network) with a convolution kernel of k×1, that is, a CNN with a convolution kernel height of k and a convolution kernel width of 1, to process it. By using different periods p, patterns of signals with different periods can be obtained. In addition, a multi-size discriminator may also include several sub-discriminators, each of which is processed using average pools of different lengths, and then CNN is used to discriminate the signal input to the discriminator model.

[0075] S202: The discriminator discriminates the input signal.

[0076] In implementation, the discriminator model can discriminate the output signal of the separator from two different angles based on a multi-cycle discriminator and a multi-scale discriminator, and can discriminate a pure human voice signal as true and an impure human voice signal as false.

[0077] S300, based on the current separator model loss and the current discriminator model loss, performing alternating adversarial training on the separator model and the discriminator model to be trained.

[0078] In implementation, the loss of the separator model can be composed of three parts, including Generative Adversarial Loss (GANLoss), Mel-Spectrogram Loss and Feature Match Loss. Specifically, the above losses can be calculated using the following loss functions:

[0079]

[0080]

[0081]

[0082] Among them, S represents the separator, D represents the discriminator, x0 represents the target signal of the separator, and x mix represents the input signal of the separator, χ(.) represents the operation of converting the time domain signal into Mel spectrum, T represents the number of layers in the discriminator to extract features, and D i represents the features in the i-th layer of the discriminator, N i represents the number of features in the i-th layer of the discriminator.

[0083] In implementation, the loss of the discriminator model can be calculated using the following loss function:

[0084]

[0085] Based on the above loss function, the discriminator is further decomposed into sub-discriminators, and the overall losses of the discriminator model and the separator model can be expressed as:

[0086]

[0087]

[0088] Among them, D j represents the j-th sub-discriminator in the multi-cycle discriminator and the multi-scale discriminator, λ mel and λ fm represents an adjustable hyperparameter.

[0089] In implementation, for ease of understanding, the alternating adversarial training of the separator model and the discriminator model to be trained can be specifically performed as follows:

[0090] The first time, the mixed music signal obtained from the training data set is input into the separator model. After the vocal signal is output, the vocal signal output by the separator model and the target signal (i.e., the real pure vocal signal) can be directly input into the discriminator model to train the initial discriminator model. The discriminator model is required to judge the vocal signal as false and the target signal as true.

[0091] After that, a new mixed signal is obtained from the training data set, the mixed music signal is input into the separator model, and the discriminator model is used to discriminate the vocal signal output by the separator model and the target signal corresponding to the mixed signal, and the loss of the separator model is calculated. Then, we can use the back-propagation algorithm to It propagates from the output layer of the separator model to the input layer, thereby updating the parameters of the separator model so that the human voice signal it outputs can be judged as true by the discriminator model.

[0092] During the next training, the loss of the discriminator model can be calculated Then use the back propagation algorithm to Propagate from the output layer of the discriminator model to the input layer, thereby updating the parameters of the discriminator model so that it can distinguish the human voice signal and the target signal output by the separator model.

[0093] In this way, the separator model and the discriminator model can continuously update their model parameters after multiple alternating iterative trainings, so that their respective loss functions converge, thereby constructing an accurate separator model structure. mix The final vocal signal x1 can be extracted by inputting the trained separator model. After that, the mixed music signal x mix Subtract the final vocal signal x1 to get the final accompaniment signal x2.

[0094] Of course, in another embodiment, the above-mentioned first output signal may also be an accompaniment signal, and the target signal corresponding to the first output signal may be a pure accompaniment signal obtained from the training database. In addition, the above-mentioned first output signal may also be a signal of a certain musical instrument in the accompaniment signal, and the target signal corresponding to the first output signal may be a pure musical instrument signal obtained from the training database. In other words, the signal type of the first output signal to be extracted may be determined by the target signal, and after the separator model and the judger model are subjected to alternating adversarial training, the actual mixed signal to be separated is input into the trained separator model to extract the first extracted signal that is ultimately required. Afterwards, the first extracted signal extracted by the separator model is subtracted from the mixed music signal to obtain the other signals that are ultimately required. The specific implementation process can be found in the above-mentioned embodiment, and this application will not be elaborated herein.

[0095] It is worth mentioning that in addition to the separator model specifically adopting the U-shaped residual network model and the discriminator model specifically adopting the HIFIGAN discriminator model, the separator model and the discriminator model can also adopt other network models to form a generative adversarial network, and this application does not impose any restrictions on this.

[0096] In one embodiment, after the separator model training is completed, step S410 may specifically include steps S411-S414, and step S420 may specifically include steps S421-S423.

[0097] S411, input the mixed music signal x into the separator model mix , and for the mixed music signal x mix Perform short-time Fourier transform analysis to obtain the complex spectrum of the mixed signal.

[0098] S412, obtain the amplitude spectrum X of the mixed signal according to the complex spectrum diagram mix and the original phase

[0099] S413, the amplitude spectrum X mix Enter the separator in the separator model.

[0100] S414, the separator outputs a mask matrix M.

[0101] S421, use the mask matrix M and the amplitude spectrum X of the original signal mix The output amplitude spectrum is obtained by point multiplication, that is, X1 = X mix ·M

[0102] S422, using the output amplitude spectrum X1 and the original phase The composed complex spectrum is used as the output complex spectrum of the separator.

[0103] S423, performing short-time inverse Fourier transform on the output complex spectrum to obtain the first extracted signal x1 of the separator.

[0104] It is worth mentioning that steps S411-S414 are similar to steps S111-S114, and steps S421-S423 are similar to steps S121-S123, which are not elaborated herein in this application.

[0105] The present application combines the mask network model with the GAN network, that is, a discriminator model is added on the basis of the separator model, and the loss function can be used to perform adversarial training on the separator model and the discriminator model, so that the separator model and the discriminator model can promote each other and improve the separation performance of the separator model. Thus, the ideal output signal is extracted based on the separator model that has completed the training. It can be understood that when the present application is used to separate the human voice and accompaniment of the mixed music signal, based on the alternating adversarial training of the separator model and the discriminator model, the separator model and the discriminator model promote and game each other, and the separator model after completing the adversarial training can output a signal close to the target signal (such as a pure human voice signal) as much as possible, so that the discriminator model is difficult to judge the signal output by the separator model as false. Therefore, after the mixed music signal is input into the separator model that completes the adversarial training, a relatively pure human voice signal can be output, which improves the separation effect of the signal extraction model for human voice and accompaniment.

[0106] Based on the same technical concept, the present application also provides a signal extraction device, see Figure 5 , the signal extraction device may include:

[0107] The separation module is used to input the mixed signal to be separated into the separator model to extract the first extracted signal; and is also used to input the first mixed signal into the separator model to be trained to obtain the first output signal;

[0108] A loss calculation module, used for inputting the first output signal and the target signal into the discriminator model for discrimination, and calculating the current separator model loss and the current discriminator model loss;

[0109] The adversarial training module is used to perform alternating adversarial training on the separator model and the discriminator model based on the current separator model loss and the current discriminator model loss.

[0110] In one embodiment, the separator model is a U-shaped residual network model, and the discriminator model is a HIFIGAN discriminator model.

[0111] In one embodiment, the adversarial training module is specifically used to:

[0112] When the current discriminator model loss is calculated, the model parameters of the separator model are fixed, and the current discriminator model loss is back-propagated to the discriminator model to adjust the model parameters of the discriminator model;

[0113] When the current separator model loss is calculated, the model parameters of the discriminator model are fixed, and the current separator model loss is back-propagated to the separator model to adjust the model parameters of the separator model.

[0114] In one embodiment, the separation module is further configured to:

[0115] The first extracted signal is subtracted from the mixed signal to be separated to extract the second extracted signal.

[0116] In one embodiment, the separation module is specifically used for:

[0117] After data preprocessing, the mixed signal to be separated is input into the separator model to obtain a mask matrix;

[0118] After the mask matrix is ​​transformed corresponding to the data preprocessing, a first extracted signal is obtained.

[0119] In one embodiment, the separation module is specifically used for:

[0120] Perform short-time Fourier transform on the mixed signal to be separated to obtain a complex frequency spectrum of the mixed signal;

[0121] Calculate the amplitude spectrum and original phase of the mixed signal to be separated based on the complex frequency spectrum of the mixed signal to be separated;

[0122] The amplitude spectrum of the mixed signal to be separated is input into the separator model to obtain a mask matrix.

[0123] In one embodiment, the separation module is further specifically used for:

[0124] Perform a dot multiplication of the amplitude spectrum of the mixed signal to be separated and the mask matrix to obtain the output amplitude spectrum;

[0125] Combine the output amplitude spectrum and the original phase to obtain the output complex spectrum;

[0126] Performing short-time inverse Fourier transform on the output complex spectrum to obtain a first extracted signal.

[0127] In one embodiment, the loss calculation module is further used to:

[0128] Calculate the loss difference between the current separator model loss and the last calculated historical separator model loss;

[0129] When the loss difference is not higher than the preset difference, it is determined that the separator model completes the adversarial training.

[0130] It should be noted that the signal extraction device provided in the above embodiment only uses the division of the above functional modules as an example when executing the signal extraction method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the signal extraction device provided in the above embodiment belongs to the same concept as the signal extraction method embodiment. The implementation principle and the technical effect to be achieved can be referred to the method embodiment, and this application will not repeat them here.

[0131] Based on the same technical concept, the present application also provides an electronic device, see Figure 6 , the electronic device comprises: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the signal extraction method as described in any of the above embodiments. The implementation principle of the electronic device for signal extraction and the technical effect to be achieved can be referred to the method embodiment, and this application will not be repeated here.

[0132] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the software product of the method for extracting human voice and accompaniment can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., including storing a number of instructions for enabling an electronic device to execute the method for extracting human voice and accompaniment described in each embodiment or some parts of the embodiment.

[0133] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A signal extraction method, characterized in that: The signal extraction method comprises: Inputting the mixed signal to be separated into the separator model to extract the first extracted signal; The separator model is obtained based on the following training steps: Inputting a first mixed signal into the separator model to obtain a first output signal; Input the first output signal and the target signal into the discriminator model for discrimination, and calculate the current separator model loss and the current discriminator model loss; Based on the current separator model loss and the current discriminator model loss, the separator model and the discriminator model are alternately trained against each other.

2. The signal extraction method according to claim 1, characterized in that: The separator model is a U-shaped residual network model, and the discriminator model is a HIFIGAN discriminator model.

3. The signal extraction method according to claim 1, characterized in that: The performing alternating adversarial training on the separator model and the discriminator model based on the current separator model loss and the current discriminator model loss comprises: When the current separator model loss is calculated, the model parameters of the discriminator model are fixed, and the current separator model loss is back-propagated to the separator model to adjust the model parameters of the separator model; When the current discriminator model loss is calculated, the model parameters of the separator model are fixed, and the current discriminator model loss is back-propagated to the discriminator model to adjust the model parameters of the discriminator model.

4. The signal extraction method according to claim 1, characterized in that: After the mixed signal to be separated is input into the separator model to extract the first extracted signal, the signal extraction method further includes: The first extracted signal is subtracted from the mixed signal to be separated to extract a second extracted signal.

5. The signal extraction method according to claim 1, characterized in that: The step of inputting the mixed signal to be separated into a separator model to extract the first extracted signal comprises: Performing data preprocessing on the mixed signal to be separated and then inputting the data into the separator model to obtain a mask matrix; After the mask matrix is ​​transformed corresponding to the data preprocessing, the first extracted signal is obtained.

6. The signal extraction method according to claim 5, characterized in that: The step of performing data preprocessing on the mixed signal to be separated and inputting the data into the separator model to obtain a mask matrix comprises: Performing short-time Fourier transform on the mixed signal to be separated to obtain a complex frequency spectrum of the mixed signal to be separated; Calculating the amplitude spectrum and the original phase of the mixed signal to be separated based on the complex frequency spectrum of the mixed signal to be separated; The amplitude spectrum of the mixed signal to be separated is input into the separator model to obtain the mask matrix.

7. The signal extraction method according to claim 6, characterized in that: After performing the transformation corresponding to the data preprocessing on the mask matrix, obtaining the first extracted signal comprises: Performing a dot multiplication of the amplitude spectrum of the mixed signal to be separated and the mask matrix to obtain an output amplitude spectrum; Combining the output amplitude spectrum with the original phase to obtain an output complex spectrum; Performing short-time inverse Fourier transform on the output complex spectrum to obtain the first extracted signal.

8. The signal extraction method according to claim 1, characterized in that: The training step of the separator model also includes: Calculating the loss difference between the current separator model loss and the last calculated historical separator model loss; When the loss difference is not higher than a preset difference, it is determined that the separator model completes the adversarial training.

9. A signal extraction device, characterized in that: The signal extraction device comprises: A separation module, used to input the mixed signal to be separated into a separator model to extract a first extracted signal; and also used to input the first mixed signal into the separator model to obtain a first output signal; A loss calculation module, used for inputting the first output signal and the target signal into the discriminator model for discrimination, and calculating the current separator model loss and the current discriminator model loss; An adversarial training module is used to perform alternating adversarial training on the separator model and the discriminator model based on the current separator model loss and the current discriminator model loss.

10. An electronic device, characterized in that: The electronic device comprises: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the signal extraction method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the signal extraction method according to any one of claims 1 to 8.

Images