Voice separation device and method, storage medium, and computer equipment

By using an encoder to extract complex features and combining CNN and RNN for speech signal separation in speech separation, the problem of low recognition rate in the prior art under multiple speakers or noise interference is solved, and higher accuracy and real-time Performance, suitable for scenarios with limited resources.

CN114743561BActive Publication Date: 2024-11-08GUANGZHOU SIXIN ELECTRONIC TECH CO LTD
View PDF 12 Cites 0 Cited by

Patent Information

Application Number
CN202210485440.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-06
Publication Date
2024-11-08
Estimated Expiration
2042-05-06

AI Technical Summary

Technical Problem

The existing voice separation technology has low recognition rate when facing multiple speakers or noise interference, and cannot meet the requirements, especially in scenarios with limited real-time applications and insufficient performance.

Method used

The encoder is used to extract the complex characteristics of the speech signal, including amplitude and phase characteristics, and perform complex operations through the separation network, combining CNN and RNN, and multi-stage downspeed processing is used to realize the separation and recovery of the speech signal.

Benefits of technology

Improves the accuracy and real-time performance of speech separation, suitable for applications with limited resource and power consumption, and reduces latency and computing resource requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114743561B_ABST
    Figure CN114743561B_ABST
Patent Text Reader

Abstract

A speech separation device and method, storage medium, and computer equipment, wherein the speech separation device includes: an encoder for encoding an input speech signal to obtain complex features of the input speech signal, wherein the complex features include amplitude features and phase features, and the input speech signal includes speech signals of one or more pronunciation objects; a separation network for performing complex operations to separate target features associated with each pronunciation object from the complex features; and a decoder for performing inverse encoding operations on the target features associated with each pronunciation object to obtain the speech signals of each pronunciation object. Thus, the accuracy of the speech separation scheme can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of speech signal processing, and in particular to a speech separation device and method, a storage medium, and a computer device. Background Art

[0002] During voice communication, in addition to the main speaker, the collected audio signal also contains interference from other speakers and noise. Current speech recognition technology can already recognize a person's words with high accuracy, but when the collected audio signal contains two or more speakers, or there is a lot of noise interference, the speech recognition rate will be greatly reduced. This problem is called the cocktail party problem. Usually we use speech separation technology to solve the cocktail party problem. The goal of speech separation is to separate the target speech (such as the speech from the target speaker) from the collected audio signal to eliminate the influence of other speakers or noise interference.

[0003] Traditional speech separation methods, such as those implemented by independent component analysis (ICA) and non-negative matrix factorization (NMF), cannot achieve good performance. In recent years, with the popularity of neural networks, attempts to use neural networks for speech separation have emerged, and experiments have shown that neural networks have better results than other alternatives.

[0004] However, the inventors have found through research that the accuracy of existing speech separation solutions still cannot meet the requirements. Summary of the invention

[0005] The technical problem solved by the present invention is how to improve the accuracy of the speech separation solution.

[0006] To solve the above technical problems, an embodiment of the present invention provides a speech separation device, comprising: an encoder, used to perform encoding operations on an input speech signal to obtain complex features of the input speech signal, wherein the complex features include amplitude features and phase features, and the input speech signal includes speech signals of one or more pronunciation objects; a separation network, used to perform complex operations to separate target features associated with each pronunciation object from the complex features; and a decoder, used to perform inverse encoding operations on the target features associated with each pronunciation object to obtain the speech signal of each pronunciation object.

[0007] Optionally, the separation network includes: a first fully connected layer, used to classify the complex features to obtain a first intermediate feature; a bidirectional convolutional network, used to obtain a convolution result based on the first intermediate feature, the convolution result including a first correlation feature and a second correlation feature, the first correlation feature representing the correlation feature between the input speech signal at a first time and the input speech signal before the first time, and the second correlation feature representing the correlation feature between the input speech signal at the first time and the input speech signal after the first time; a recurrent network, used to obtain a recurrent result based on the first correlation feature and the second correlation feature, the recurrent result including the correlation feature between the input speech signal at the second time and the input speech signal before the second time, the second time being later than the first time; a second fully connected layer, used to classify the third correlation feature to obtain a mask of each pronunciation object; a multiplier, used to calculate the product between the complex feature and the mask of each pronunciation object to obtain a target feature associated with each pronunciation object.

[0008] Optionally, the bidirectional convolutional network includes multiple basic modules, and the multiple basic modules are connected in series, and each basic module includes: a third fully connected layer, used to classify the input signal of the basic module to obtain a second intermediate feature; a multi-level convolution layer, used to perform multi-level convolution operations on the second intermediate feature to obtain a target convolution result; and a fourth fully connected layer, used to classify the target convolution result to obtain the output signal of the basic module.

[0009] Optionally, the multi-level convolution layer includes a first convolution layer and a multi-level second convolution layer: the first convolution layer is used to perform a convolution operation on the second intermediate feature to obtain a first convolution result; the current convolution layer is used to perform a speed-down convolution operation on the current input feature of the input to obtain an intermediate convolution feature, and perform a speed-up process on the intermediate convolution feature to obtain the output feature of the current convolution layer; wherein the current convolution layer is the second convolution layer of any level, and the input features of each second convolution layer except the first level second convolution layer include the output features of the previous level convolution layer of the current convolution layer, and the input features of the first level second convolution layer are the first convolution result.

[0010] Optionally, the intermediate convolution features are subjected to speed-up processing by a speed-up convolution operation.

[0011] Optionally, the recurrent network includes a long short-term memory network or a gated recurrent unit.

[0012] Optionally, the first fully connected layer has an activation function cPReLU, and the activation function cPReLU is used to perform the calculation of the following formula: Among them, x ris the real part of the complex characteristic, x i is the imaginary part of the complex characteristic, w r For x r The weight, w i For x i The weight of .

[0013] An embodiment of the present invention also provides a speech separation method, comprising: performing an encoding operation on an input speech signal to obtain complex features of the input speech signal, wherein the complex features include amplitude features and phase features, and the input speech signal includes speech signals of one or more pronunciation objects; performing complex operations to separate target features associated with each pronunciation object from the complex features; and performing an inverse encoding operation on the target features associated with each pronunciation object to obtain the speech signal of each pronunciation object.

[0014] An embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0015] An embodiment of the present invention further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0016] Compared with the prior art, the technical solution of the embodiment of the present invention has the following beneficial effects:

[0017] The embodiment of the present invention provides a speech separation device, comprising: an encoder, which is used to perform encoding operations on an input speech signal to obtain complex features of the input speech signal, wherein the complex features include amplitude features and phase features, and the input speech signal includes speech signals of one or more pronunciation objects; a separation network, which is used to perform complex operations to separate target features associated with each pronunciation object from the complex features; and a decoder, which is used to perform inverse encoding operations on the target features associated with each pronunciation object to obtain the speech signals of each pronunciation object. In the speech separation device of the embodiment of the present invention, the encoder extracts the complex features of the input speech signal, and the complex features include amplitude information and phase information in the input speech signal. Complex operations are used in both the separation network and the decoder, which can retain more information in deep learning, thereby improving the accuracy of speech separation. If the speech separation device is applied in a speech communication system, the performance of the speech communication system can be effectively improved.

[0018] Furthermore, compared with the existing Conv-TasNet, the speech separation device provided by the embodiment of the present invention has the following advantages: Advantage 1, because the separation network adopts CNN and RNN at the same time, and each convolution layer adopts complex convolution operation, satisfactory performance can be obtained with low delay, and real-time applications can be supported; Advantage 2, CNN and RNN are adopted at the same time, and multi-level speed reduction processing is added, which requires less resources for implementation and can be suitable for applications with limited resources and power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of the structure of a speech separation device according to an embodiment of the present invention;

[0020] Figure 2 for Figure 1 A schematic diagram of a specific implementation of the encoder;

[0021] Figure 3 for Figure 1 A schematic diagram of a specific implementation of the decoder;

[0022] Figure 4 for Figure 1 A schematic diagram of a specific implementation of a separation network;

[0023] Figure 5 for Figure 4 A schematic diagram of a specific implementation of a bidirectional convolutional network;

[0024] Figure 6 for Figure 5 A schematic diagram of a specific implementation of a single basic module;

[0025] Figure 7 Schematic diagram of the structure of MrCNN according to an embodiment of the present invention;

[0026] Figure 8 A schematic diagram of a complex convolution operation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] As mentioned in the background art, there is a need to improve the accuracy of speech separation solutions.

[0028] Specifically, the prior art provides a fully convolutional time-domain audio separation network Conv-TasNet, which is a deep learning speech separation architecture based on convolutional neural networks (CNN). This architecture mainly includes three parts: encoder, separation network or separation unit (Separator) and decoder.

[0029] The encoder is used to calculate the features of the input mixed speech signal. The encoder consists of a single one-dimensional reduced-speed convolution layer (represented by 1D-Conv), which is used to perform a reduced-speed convolution operation on the input mixed speech signal.

[0030] The separation network is used to calculate the ratio of the eigenvalues ​​of each speech signal to the eigenvalues ​​of the mixed speech signal, that is, the mask value. The separation network includes multiple layers of 1D-Conv, nonlinear excitation function (represented by PReLU) and S-type activation function (also known as Sigmoid activation function). In order to support the long delay effect, these 1D-Convs are all dilated CNNs, and the dilated values ​​of the 1D-Convs of each layer can be set to 1, 2, ..., 2k in sequence, where k is a positive integer.

[0031] The decoder corresponds to the encoder and is used to perform the inverse operation of the encoder to restore the time domain data of the corresponding speech signal according to the features of each separated speech signal. The decoder can be composed of a one-dimensional speed-up 1D-Conv, which is used to perform a speed-up convolution operation on the features of each speech signal.

[0032] However, the inventors found through research that although Conv-TasNet greatly improves the performance of speech separation, it still has the following three shortcomings:

[0033] Disadvantage 1: Not suitable for real-time applications. In real-time communication scenarios, the delay of speech processing is generally required to be no more than 0.1 seconds. However, in order to obtain acceptable speech separation performance, the convolution delay of Conv-TasNet needs to be more than 0.3 seconds.

[0034] Disadvantage 2: It requires more processing resources, which is not conducive to applications with limited resources and power consumption, such as mobile phones and other terminal devices. Although Conv-TasNet uses hollow CNN to reduce the resources used in order to support long delay effects, in practical applications, Conv-TasNet still requires larger computing resources.

[0035] Disadvantage 3: The speech separation performance of Conv-TasNet still needs to be improved. Generally speaking, Conv-TasNet only studies the amplitude information of the input mixed speech signal, and the performance of the Conv-TasNet trained in this way is poor. For this reason, the phase information of the input mixed speech signal can be introduced into a new Conv-TasNet. However, since Conv-TasNet is still a real-valued network, its speech separation performance cannot reach the optimal level.

[0036] To solve the above problems, an embodiment of the present invention provides a speech separation device, which may include: an encoder, used to perform encoding operations on an input speech signal to obtain complex features of the input speech signal, wherein the complex features include amplitude features and phase features, and the input speech signal includes speech signals of one or more pronunciation objects; a separation network, used to perform complex operations to separate target features associated with each pronunciation object from the complex features; and a decoder, used to perform inverse encoding operations on the target features associated with each pronunciation object to obtain the speech signals of each pronunciation object. This can improve the accuracy of speech separation.

[0037] In order to make the above-mentioned objects, features and beneficial effects of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0038] See also Figure 1 , Figure 1 Schematic diagram of the structure of a speech separation device 10 according to an embodiment of the present invention. The speech separation device 10 can be used to realize single-channel speech separation, that is, to realize speech separation of sound signals collected by a single speech collection device (such as a microphone, etc.). The speech separation device 10 can be realized by hardware such as an integrated circuit (IC), and the speech separation device 10 can be applied to devices such as telephones, mobile phones, computers, hearing aids, etc. that require audio signal processing. The speech separation device 10 includes an encoder 101, a separation network 102, and a decoder 103, which are described in detail below.

[0039] The encoder 101 is used to perform encoding operation on the input speech signal to obtain complex features of the input speech signal, wherein the complex features include amplitude features and phase features, and the input speech signal includes speech signals of one or more pronunciation objects.

[0040] The input voice signal is a mixed audio containing sounds emitted by multiple pronunciation objects. For example, the input voice signal can be a sound signal collected by a sound collection device. The pronunciation object refers to the object that emits the sound signal. For example, when the input voice signal includes the voices of one or more people, the pronunciation object refers to the one or more people. The pronunciation object can also refer to other creatures or objects that can emit sound signals, such as animals and car horns.

[0041] The real part of the complex feature represents the amplitude feature of the input speech signal, and the imaginary part of the complex feature represents the phase feature of the input speech signal.

[0042] In a specific embodiment, see Figure 2 , Figure 2 for Figure 1Schematic diagram of a specific implementation of the encoder 101 in FIG. The encoder 101 may include two parallel one-dimensional speed reduction convolution layers ( Figure 2 In the figure, “1D-Conv(↓N)” is used, where N is the multiple of the speed reduction and is a positive integer) 1011, 1012 and two activation functions (ReLU) 1013, 1014. The output of 1D-Conv 1011 is used as the input of the activation function (ReLU) 1013. After the input speech signal passes through 1D-Conv 1011 and the activation function 1013, the amplitude feature of the input speech signal can be obtained, that is, the real part of the complex feature (i.e. Figure 2 After the input speech signal passes through 1D-Conv1012 and the excitation function 1014, the phase feature of the input speech signal can be obtained, that is, the imaginary part of the complex feature (i.e. Figure 2 "Complex Characteristics (Imaginary Part)" in .

[0043] Optionally, the input speech signal can be a one-dimensional signal, and the data rate of the input speech signal is recorded as fs (e.g., 16Ksps). The complex signal output by the encoder 101 includes M complex features, and the complex signal including M complex features can also be called an M-dimensional complex signal; the data rate of the complex features can be expressed as fs / N, where N represents the multiple of the speed reduction. Among them, M and N are positive integers, and the value of M is usually 2, 4, 8...

[0044] The separation network 102 is used to perform complex operations to separate target features associated with each pronunciation object from the complex features.

[0045] Among them, complex operations can include complex addition operations, complex multiplication operations, complex convolution operations, etc. Complex features contain real parts and imaginary parts. When the separation network performs various operations on complex features, it should be considered to perform each calculation step according to the complex operation rules.

[0046] The target feature associated with each pronunciation object refers to the feature of the speech signal emitted by each pronunciation object. The target feature is also a complex number, that is, the target feature includes a real part and an imaginary part. Correspondingly, the real part of the target feature represents the amplitude feature of the speech signal emitted by each pronunciation object, and the imaginary part of the target feature represents the phase feature of the speech signal emitted by each pronunciation object.

[0047] The decoder 103 is used to perform an inverse encoding operation on the target features associated with each pronunciation object to obtain the speech signal of each pronunciation object. The decoder 103 is used to perform the inverse operation of the encoder 101 and restore the time domain data of the speech according to the features of each separated speech. Optionally, the decoder 103 can output the speech signals of n pronunciation objects, where n is a natural number.

[0048] In a specific embodiment, see Figure 3 , Figure 3 for Figure 1 A schematic diagram of a specific implementation of the decoder 103 is shown in FIG. 1 . The decoder 103 may include two one-dimensional speed-up convolution layers 1031 and 1032 (represented as "1D-Conv(↑N)" in the figure) and an adder 1033, where N represents the speed-up multiple, and the speed-up multiple of the convolution layer in the decoder 103 is equal to the speed-down multiple of the encoder 101. The real part of the target feature associated with each pronunciation object passes through the one-dimensional speed-up convolution layer 1031, and the imaginary part of the target feature associated with each pronunciation object passes through the one-dimensional speed-up convolution layer 1032. The output signals of the one-dimensional speed-up convolution layer 1031 and the one-dimensional speed-up convolution layer 1032 are passed through the adder 1033 to obtain the speech signal of this pronunciation object.

[0049] Optionally, when the complex feature output by the encoder 101 is an M-dimensional complex signal, the target feature associated with each pronunciation object is also an M-dimensional complex signal, and the data rate of the target feature associated with each pronunciation object is recorded as fs / N. The speech signal of each pronunciation object output is a one-dimensional signal, and the data rate is recorded as fs, where N represents the multiple of the speed increase. Wherein, M and N are positive integers.

[0050] In the above-mentioned speech separation device 10, the encoder 101 extracts the complex features of the input speech signal, which include the amplitude information and phase information in the input speech signal. Complex operations are used in the separation network 102 and the decoder 103, which can retain more information in deep learning, thereby improving the accuracy of speech separation. If the speech separation device 10 is applied to a speech communication system, the performance of the speech communication system can be effectively improved.

[0051] In one embodiment, see Figure 4 , Figure 4 for Figure 1 Schematic diagram of a specific implementation of the separation network 102. The separation network 102 may include: a first fully connected layer (Fully Connected layers, referred to as FC) 1021, a bidirectional convolutional network (denoted by CNN in the figure) 1022, a recurrent network (denoted by RNN in the figure) 1023, a second fully connected layer (denoted by FC in the figure) 1024 and a multiplier 1025.

[0052] Among them, the first fully connected layer 1021 is used to classify the complex features to obtain first intermediate features; the bidirectional convolutional network 1022 is used to obtain a convolutional result according to the first intermediate features, and the convolutional result includes a first correlation feature and a second correlation feature. The first correlation feature represents the correlation feature between the input speech signal at the first time and the input speech signals before the first time, and the second correlation feature represents the correlation feature between the input speech signal at the first time and the input speech signals after the first time; the recurrent network 1023 is used to obtain a recurrent result according to the convolutional result, and the recurrent result includes the correlation feature between the input speech signal at the second time and the input speech signals before the second time, and the second time is later than the first time; the second fully connected layer 1024 is used to classify the recurrent result to obtain the mask (Mask) of each pronunciation object; the multiplier 1025 is used to calculate the product between the complex features and the masks of each pronunciation object to obtain the target features associated with each pronunciation object. Further, the multiplier 1025 corresponds to the pronunciation object, and each multiplier 1025 obtains the target features associated with the corresponding pronunciation object.

[0053] The following will elaborate on each part of the separation network 102.

[0054] Two FCs (including FC1021 and FC1024) can be used to change the dimension of the data input to the separation network 102. FC1021 changes the data dimension of the complex features from M to K; FC1024 changes the data dimension back from K to M. Usually, K < M, that is, the data dimension is reduced by FC1021 to reduce the computational amount of the bidirectional convolutional network 1022 and the recurrent network 1023. The operation equation of FC1021 can be expressed as y = Wx + b. Among them, the input data x of FC1021 ∈ C M , the weight parameter W ∈ C K×M , b ∈ C K ; the output data of FC1021 (i.e., the first intermediate features) y ∈ C K , C represents complex features, C M represents M-dimensional complex features, C K represents K-dimensional complex features. The equation of FC1024 corresponds to the operation equation of FC1021 and will not be elaborated here.

[0055] The bidirectional convolution network 1022 can be implemented by CNN, and CNN can include multiple convolution layers. The complex features of the input speech signal are input into CNN through FC1021, and the convolution results of each point in the complex features are obtained through multiple convolution layers. Since the complex features include the correlation between the points in the input speech signal, the convolution results obtained through the multiple convolution layers include the bidirectional correlation between the points in time. The bidirectional correlation in time includes the correlation between the current time and the past (recorded as the first correlation feature) and the correlation between the current time and the future (recorded as the second correlation feature).

[0056] The recurrent network 1023 can be implemented by a unidirectional recurrent neural network (RNN), or the recurrent network 1023 can be implemented by a bidirectional recurrent neural network (RNN). Optionally, the recurrent network 1023 may include a long short-term memory network (LSTM) or a gated recurrent unit (GRU), or may include other commonly used RNNs, which are not described here. Among them, the unidirectional RNN is used to extract the unidirectional association of each point in time, which is represented by the association feature between the input speech signal at the second time and the input speech signal before the second time.

[0057] In a specific embodiment, for each multiplier 1025, a complex multiplication calculation can be performed on the complex feature and the mask of a single pronunciation object to obtain the target feature associated with this pronunciation object.

[0058] In another specific embodiment, for each multiplier 1025, the real part of the complex feature and the real part of the mask of a single pronunciation object can be multiplied to obtain a first result, and then the imaginary part of the complex feature and the imaginary part of the mask of this pronunciation object can be multiplied to obtain a second result, and finally the sum of the first result and the second result is calculated to obtain the output data of this multiplier 1025.

[0059] It should be noted that if only the CNN in the above convolutional network 1022 is relied upon, since the CNN needs to process the association between the current time and the future, the bidirectional convolutional network 1022 must adopt a delay unit, resulting in a large delay in the speech separation solution, which also causes the disadvantage 1 of the aforementioned Conv-TasNet. And if only relying on CNN to achieve efficient speech separation effect, the number of levels of convolutional layers that need to be deployed in CNN is huge, and a large amount of resources need to be consumed. Therefore, the embodiment of the present invention adds RNN (that is, recurrent network 1023) after CNN. Compared with the bidirectional RNN, the unidirectional RNN extracts the unidirectional association of each point in time, so the amount of data processed by the unidirectional RNN is less than that of CNN or bidirectional RNN, and the delay is also smaller. However, compared with the bidirectional RNN, the accuracy of speech separation may be affected by the unidirectional RNN because it processes less data.

[0060] Optionally, the first fully connected layer (i.e., FC1021) has an activation function (denoted by cPReLU in the figure) 1026, and the activation function 1026 is used to perform the calculation of the following formula (1):

[0061]

[0062] Among them, x r is the real part of the complex characteristic, x i is the imaginary part of the complex characteristic, w r For x r The weight, w i For x i The weight of w r and w i It has a preset initial value and is continuously iterated during the training process of the separation network 102 .

[0063] In one embodiment, Figure 4 The positions of the recurrent network 1023 and the bidirectional convolutional network 1022 in the separation network 102 can be interchanged, that is, the first intermediate feature is first input into the recurrent network 1023, and the output of the recurrent network 1023 is used as the input of the bidirectional convolutional network 1022. Figure 3 In the separation network 102 of this embodiment, the first intermediate feature is first extracted from the unidirectional or bidirectional temporal correlation of each point after passing through the unidirectional or bidirectional RNN.

[0064] In one embodiment, see Figure 5 and Figure 6 , Figure 5 for Figure 4 A schematic diagram of a specific implementation of the bidirectional convolutional network 1022, Figure 6 for Figure 5Schematic diagram of a specific implementation of a single basic module. The bidirectional convolutional network 1022 may include a plurality of basic modules (Unit Block, abbreviated as UBlock), and the plurality of basic modules are connected in series. Each basic module may include: a third fully connected layer ( Figure 6 denoted as "FC601" in ), for classifying the input signal of the basic module to obtain a second intermediate feature; a multi-level convolutional layer 602, for performing multi-level convolutional operations on the second intermediate feature to obtain a target convolutional result; a fourth fully connected layer ( Figure 6 denoted as "FC603" in ), for classifying the target convolutional result to obtain the output signal of the basic module.

[0065] FC601 and FC603 can be used to change the dimension of the data input to this basic module. FC601 changes the data dimension from M to K; FC603 changes the data dimension back from K to M. Usually K < M, that is, the data dimension is reduced by FC603 to reduce the computational amount of the multi-level convolutional layer 602. Other descriptions of FC601 and FC603 can be found in Figure 4 the relevant descriptions of FC1021 and FC1024 in, which will not be elaborated here.

[0066] Optionally, FC601 has an activation function cPReLU6011, and FC603 has an activation function cPReLU6031. The cPReLU6011 and cPReLU6031 can be used to perform complex number operations, and the calculation formula can be seen in the above formula (1).

[0067] The multi-level convolutional layer 602 may include a Multi-Rate CNN, abbreviated as MrCNN. The multi-level convolutional layer includes a first convolutional layer and n second convolutional layers, where n is a positive integer. The first convolutional layer is used to perform a convolutional operation on the second intermediate feature to obtain a first convolutional result; each second convolutional layer is used to perform a decimation convolutional operation on the current input feature to obtain an intermediate convolutional feature, and perform an interpolation process on the intermediate convolutional feature to obtain the output feature of the current convolutional layer; wherein, the current convolutional layer is any one of the second convolutional layers, and the input feature of each second convolutional layer except the first second convolutional layer includes the output feature of the previous convolutional layer of the current convolutional layer, and the input feature of the first second convolutional layer is the first convolutional result. The decimation multiple and the interpolation multiple of each second convolutional layer are the same. For example, first perform 2-fold decimation and then 2-fold interpolation.

[0068] In this embodiment, the first convolution layer is not decelerated, so more information can be retained. The second convolution layer at each level is decelerated, that is, some of the data is discarded, which can reduce the amount of computation during convolution, reduce the amount of computation, and obtain sufficiently deep time correlation operations to solve the delay problem.

[0069] In a specific embodiment, see Figure 7 , Figure 7 It is a structural diagram of a MrCNN according to an embodiment of the present invention. The input of the MrCNN is the second intermediate feature. The first convolution layer may include a one-dimensional convolution network (1D-Conv) 701, the first-level second convolution layer may include a one-dimensional speed-down convolution network (indicated by "1D-Conv(↓2)" in the figure) 7021 and a speed-up convolution network 7031 (indicated by "1D-Conv(↑2)" in the figure), the second-level second convolution layer may include a one-dimensional speed-down convolution network 7022 and a speed-up convolution network 7032, ..., the n-th-level second convolution layer may include a one-dimensional speed-down convolution network 702n and a speed-up convolution network 703n. The data output by each convolution layer (including the first convolution layer and the second convolution layers at each level) are complex-added by an adder 704. Optionally, the data output by each convolution layer is added and then passed through an excitation function cPReLU705 to obtain a target convolution result. Figure 7 Each convolutional network is a one-dimensional channel-by-channel (depth-wise) convolution operation unit. The "2" in 1D-Conv(↑2) and 1D-Conv(↓2) indicates that the speed-up or speed-down factor is 2 times, which can also be changed to other factors.

[0070] It should be noted that each speed-up convolutional network (7031, 7032, ..., 703n) can also be replaced by a module that only performs speed-up processing. If the speed-up processing is performed directly, signal distortion may be caused due to signal interference, and using the speed-up convolutional network for speed-up can effectively solve this problem.

[0071] For each convolution network (or called convolution layer or convolution unit) in the embodiment of the present invention, complex convolution can be performed, see Figure 8 , Figure 8It is a schematic diagram of a complex convolution operation of an embodiment of the present invention. The real part and imaginary part of the input complex signal are obtained respectively, and are recorded as input (real part) and input (imaginary part). When performing complex convolution, the convolution layer whose input parameter of the input (real part) is a real number is convolved to obtain the first result, the convolution layer whose input parameter of the input (real part) is an imaginary number is convolved to obtain the second result, the convolution layer whose input parameter of the input (imaginary part) is a real number is convolved to obtain the third result, and the convolution layer whose input parameter of the input (imaginary part) is an imaginary number is convolved to obtain the fourth result. The sum of the first result and the fourth result is calculated as the real part of the output complex signal, which is recorded as output (real part). The sum of the second result and the third result is calculated as the imaginary part of the output complex signal, which is recorded as output (imaginary part). Figure 8 Each convolution layer in may include a 1-dimensional complex convolution network (represented by 1D-rconv in the figure). Compared with the existing Conv-TasNet, the speech separation device provided by the embodiment of the present invention has the following advantages:

[0072] Advantage 1: Support for real-time applications. Because the separation network uses both CNN and RNN, and each convolution layer uses complex convolution operations, satisfactory performance can be obtained with low latency. Specifically, the delay of this speech separation device does not exceed 0.1 seconds, and with a total parameter volume of 500K, the following performance can be obtained: the signal to distortion ratio (Source to Distortion ratio, referred to as SDR) can reach 11.7 decibels (dB), and the perceptual evaluation of speech quality (Perceptual Evaluation of Speech Quality, referred to as PESQ) is approximately 2.5. In other words, the performance of this speech separation device is better.

[0073] Advantage 2: It uses both CNN and RNN and adds multi-level speed reduction processing, which requires fewer resources to implement and can be suitable for applications with limited resources and power consumption.

[0074] An embodiment of the present invention also provides a speech separation method, comprising: performing an encoding operation on an input speech signal to obtain complex features of the input speech signal, wherein the complex features include amplitude features and phase features, and the input speech signal includes speech signals of one or more pronunciation objects; performing complex operations to separate target features associated with each pronunciation object from the complex features; and performing an inverse encoding operation on the target features associated with each pronunciation object to obtain the speech signal of each pronunciation object.

[0075] For more information about the explanation of nouns, working principles, and working methods in the speech separation method described in this embodiment, please refer to the above Figures 1 to 8The relevant description of the illustrated embodiment will not be repeated here.

[0076] Furthermore, an embodiment of the present invention also discloses a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the technical solution of the above-mentioned speech separation method is executed. The storage medium may be a computer-readable storage medium.

[0077] Furthermore, an embodiment of the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the technical solution of the above-mentioned speech separation method when running the computer program.

[0078] Specifically, in the embodiment of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0079] It should also be understood that the storage medium and memory in the embodiments of the present application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0080] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.

[0081] The "plurality" appearing in the embodiments of the present application refers to two or more.

[0082] The first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects. There is no order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0083] The "connection" that appears in the embodiments of the present application refers to various connection methods such as direct connection or indirect connection to achieve communication between devices, and the embodiments of the present application do not impose any limitations on this.

[0084] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A speech separation device, characterized in that: include: An encoder is used to perform an encoding operation on an input speech signal to obtain complex features of the input speech signal, wherein the complex features include amplitude features and phase features, and the input speech signal includes speech signals of one or more pronunciation objects; A separation network, used to perform complex operations to separate target features associated with each pronunciation object from the complex features; the separation network includes: a first fully connected layer, used to classify the complex features to obtain a first intermediate feature; a bidirectional convolutional network, used to obtain a convolution result based on the first intermediate feature, the convolution result includes a first associated feature and a second associated feature, the first associated feature represents the associated feature between an input voice signal at a first time and an input voice signal before the first time, and the second associated feature represents the associated feature between an input voice signal at a first time and an input voice signal after the first time; a recurrent network, used to obtain a recurrent result based on the convolution result, the recurrent result includes the associated feature between an input voice signal at a second time and an input voice signal before the second time, the second time being later than the first time; a second fully connected layer, used to classify the recurrent result, and obtain a mask of each pronunciation object; a multiplier, used to calculate the product between the complex features and the masks of each pronunciation object, and obtain the target feature associated with each pronunciation object; The decoder is used to perform inverse encoding operations on the target features associated with each pronunciation object to obtain the speech signal of each pronunciation object.

2. The speech separation device according to claim 1, characterized in that: The bidirectional convolutional network includes a plurality of basic modules, the plurality of basic modules are connected in series, and each basic module includes: A third fully connected layer, used to classify the input signal of the basic module to obtain a second intermediate feature; A multi-level convolution layer, used for performing a multi-level convolution operation on the second intermediate feature to obtain a target convolution result; The fourth fully connected layer is used to classify the target convolution result to obtain the output signal of the basic module.

3. The speech separation device according to claim 2, characterized in that: The multi-level convolutional layer includes a first convolutional layer and a multi-level second convolutional layer: The first convolution layer is used to perform a convolution operation on the second intermediate feature to obtain a first convolution result; The current convolution layer is used to perform a speed-down convolution operation on the current input features of the input to obtain intermediate convolution features, and to perform a speed-up process on the intermediate convolution features to obtain output features of the current convolution layer; wherein the current convolution layer is the second convolution layer of any level, and the input features of each second convolution layer except the first level second convolution layer include the output features of the previous level convolution layer of the current convolution layer, and the input features of the first level second convolution layer are the first convolution results.

4. The speech separation device according to claim 3, characterized in that: The intermediate convolution features are subjected to a speed-up process by a speed-up convolution operation.

5. The speech separation device according to claim 1, characterized in that: The recurrent network includes a long short-term memory network or a gated recurrent unit.

6. The speech separation device according to claim 1 or 2, characterized in that: The first fully connected layer has an activation function cPReLU, and the activation function cPReLU is used to perform the calculation of the following formula: Among them, x r is the real part of the complex characteristic, x i is the imaginary part of the complex characteristic, w r For x r The weight, w i For x i The weight of .

7. A speech separation method, characterized in that: include: Performing an encoding operation on an input speech signal to obtain complex features of the input speech signal, wherein the complex features include amplitude features and phase features, and the input speech signal includes speech signals of one or more pronunciation objects; A separation network is used to perform complex operations, so as to separate target features associated with each pronunciation object from the complex features; the separation network includes: a first fully connected layer, which is used to classify the complex features to obtain a first intermediate feature; a bidirectional convolutional network, which is used to obtain a convolution result based on the first intermediate feature, and the convolution result includes a first associated feature and a second associated feature, wherein the first associated feature represents the associated feature between an input speech signal at a first time and an input speech signal before the first time, and the second associated feature represents the associated feature between an input speech signal at a first time and an input speech signal after the first time; a recurrent network, which is used to obtain a recurrent result based on the convolution result, and the recurrent result includes the associated feature between an input speech signal at a second time and an input speech signal before the second time, and the second time is later than the first time; a second fully connected layer, which is used to classify the recurrent result and obtain a mask of each pronunciation object; a multiplier, which is used to calculate the product between the complex feature and the mask of each pronunciation object, and obtain the target feature associated with each pronunciation object; The target features associated with each pronunciation object are inversely encoded to obtain the speech signal of each pronunciation object.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 7 are implemented.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to claim 7 are implemented.

Citation Information

Patent Citations

  • Voice separation method, voice recognition method and related equipment

    CN110459238A

  • Multi-mode voice separation method and system

    CN113035227A

  • Scene adaptive hearing aid audio enhancement system based on neural network

    CN114245280A

  • Human voice separation method for voices of multiple speakers, terminal equipment and storage medium

    CN114333852A

  • Low-latency multi-speaker speech recognition

    US20200135209A1