Vehicle-mounted multi-channel speech separation method and device, electronic equipment and storage medium

Abstract

The present application relates to a vehicle-mounted multi-channel speech separation method, device, electronic equipment and storage medium, the method comprising: determining mixed training set and test set of human speech under different space sizes and different microphone array types according to the in-vehicle space and M microphones by using the mirror source method; converting the speech of the mixed training set to the frequency domain; inputting the signal converted to the frequency domain into the network to obtain an enhanced speech signal after training; calculating a loss function using the signal converted to the frequency domain and the enhanced speech signal; updating the network weight according to the loss function; training multiple rounds, processing the test set in each round, calculating the index of the test set; selecting a model according to the index of the test set in multiple rounds; obtaining M channel frequency domain signals through the model; changing the output M channel frequency domain speech signals to the time domain to obtain separated multi-channel speech signals. The present application solves the problem of repeated training, and the mirror source method is more effective than existing methods in simulating background noise in real scenarios.

Description

Technical Field

[0001] This invention relates to the field of speech separation technology, and more specifically to in-vehicle multi-channel speech separation methods, devices, electronic equipment, and storage media. Background Technology

[0002] Currently, the main methods for vehicle-mounted multi-channel speech separation are divided into neural network-based separation algorithms and traditional array signal-based algorithms. Traditional array signal-based algorithms are easily affected by hardware structure and noise, resulting in more speech residue after processing. Neural network-based separation algorithms have a large computational load and require retraining of the network when microphone array switching occurs. When multiple people speak at the same time, residue is easily generated in channels other than the speaker's channel. When the signal-to-noise ratio is low, there is obvious noise residue in the speaker's channel. Summary of the Invention

[0003] This invention provides a method, apparatus, electronic device, and storage medium for in-vehicle multi-channel voice separation, which can solve the above-mentioned technical problems.

[0004] In a first aspect, embodiments of the present invention provide an in-vehicle multi-channel voice separation method, comprising:

[0005] The mirror source method was used to determine mixed training and test sets for human speech under different space sizes and microphone arrays based on the vehicle interior space and M microphones.

[0006] Convert the speech of the mixed training set to the frequency domain;

[0007] The signal converted to the frequency domain is input into the network and trained to obtain the enhanced speech signal.

[0008] The loss function is calculated using the signal converted to the frequency domain and the enhanced speech signal.

[0009] Update network weights based on the loss function;

[0010] The training process is repeated in multiple rounds, with the test set being processed in each round and metrics for the test set being calculated.

[0011] The model is selected based on the metrics of the test set in multiple rounds;

[0012] The frequency domain signals of M channels are obtained through the model;

[0013] The frequency domain speech signals of M channels are transformed to the time domain to obtain the final separated multi-channel speech signals;

[0014] The mixed training and test sets include: clean target speech and noise containing clean target speech.

[0015] Further, in the vehicle-mounted multi-channel speech separation method, converting the mixed training set speech into a frequency domain comprises:

[0016] Converting the mixed training set speech into a frequency domain through framing, adding a Hanning window, and short-time Fourier transform;

[0017] Wherein, the framing is 32 ms per frame, and the frame overlap rate is 50%.

[0018] Further, in the vehicle-mounted multi-channel speech separation method, inputting the converted signal into a network to obtain an enhanced speech signal comprises:

[0019] Inputting the converted signal into the network and implementing a frequency band division operation in frequency;

[0020] After the frequency band division operation, sequentially inputting into an encoder layer, an LSTM layer, and a decoder layer;

[0021] Implementing a frequency band merging operation to output M-channel frequency domain signals;

[0022] Wherein, the last layer of the decoder layer uses a full connection layer.

[0023] Further, in the vehicle-mounted multi-channel speech separation method, updating the network weight according to a loss function comprises:

[0024] Updating the network weight according to a mixed weighted loss function combining the average mean square error of M channels and the average mean square error between individual channels.

[0025] Further, in the vehicle-mounted multi-channel speech separation method, the network uses an asymmetric network.

[0026] Further, in the vehicle-mounted multi-channel speech separation method, converting the M-channel frequency domain speech signals into a time domain to obtain the final separated multi-channel speech signals comprises:

[0027] Converting the M-channel frequency domain speech signals into a time domain through inverse short-time Fourier transform, window removal, and overlap addition to obtain the final separated multi-channel speech signals.

[0028] Further, in the vehicle-mounted multi-channel speech separation method, the encoder layer uses a grouped convolution design.

[0029] In a second aspect, the embodiments of the present application further provide a vehicle-mounted multi-channel speech separation device, which comprises:

[0030] A determination module configured to determine a mixed training set and a test set of human speech in different space sizes and different microphone array types according to an in-vehicle space and M microphones using a mirror source method.

[0031] a conversion module configured to convert the mixed training set to the frequency domain;

[0032] a training module configured to input the converted signal to the network to obtain an enhanced speech signal;

[0033] a first calculation module configured to calculate a loss function by using the converted signal and the enhanced speech signal;

[0034] an updating module configured to update the network weight according to the loss function;

[0035] a second calculation module configured to train multiple rounds, process the test set in each round, and calculate an index of the test set;

[0036] a screening module configured to select a model according to the index of the test set in the multiple rounds;

[0037] an obtaining module configured to obtain the frequency domain signal of the M channels by using the model;

[0038] a changing module configured to change the frequency domain signal of the M channels to the time domain to obtain a final separated multi-channel speech signal;

[0039] The mixed training set and the test set include clean target speech and noise containing the clean target speech.

[0040] In a third aspect, an electronic device is provided, which includes a processor and a memory.

[0041] The processor is configured to execute the vehicle-mounted multi-channel speech separation method according to any one of the above aspects by calling programs or instructions stored in the memory.

[0042] In a fourth aspect, a computer readable storage medium is provided, which stores programs or instructions, and the programs or instructions make a computer execute the vehicle-mounted multi-channel speech separation method according to any one of the above aspects.

[0043] The beneficial effects of the present application are: the present application determines mixed training sets and test sets of people speaking under different space sizes and different microphone array types according to the in-vehicle space and M microphones by adopting the mirror source method; converts the speech of the mixed training set to the frequency domain; inputs the signal converted to the frequency domain into the network to obtain an enhanced speech signal after training; calculates a loss function using the signal converted to the frequency domain and the enhanced speech signal; updates the network weight according to the loss function; trains multiple rounds, processes the test set in each round, calculates the index of the test set, selects a model according to the index of the test set in multiple rounds, obtains the frequency domain signal of M channels through the model, changes the M-channel frequency domain speech signal to the time domain, and obtains the final separated multi-channel speech signal. The present application solves the problem of repeated training by feeding the data of different arrays into the training with a set prior, and the mirror source method is more effective than the existing method in simulating background noise in a real scene. Unlike the PIT training strategy used in general multi-channel separation networks, the present application uses a position approximation method corresponding to the channel to simultaneously calculate the loss function between all channels and a single channel, so that the multi-channel speech output by the network is cleaner; the present application reduces the computational amount of the network by adopting grouped convolution and an asymmetric network. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 A vehicle-mounted multi-channel speech separation method is provided for the embodiments of the present application.

[0045] Figure 2 A vehicle-mounted multi-channel speech separation device is provided for the embodiments of the present application.

[0046] Figure 3 A schematic block diagram of an electronic device is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0047] The principles and characteristics of the present application are described below in combination with the drawings, and the examples are only used to explain the present application and are not used to limit the scope of the present application.

[0048] Figure 1 A vehicle-mounted multi-channel speech separation method is provided for the embodiments of the present application.

[0049] In a first aspect, the embodiments of the present application provide a vehicle-mounted multi-channel speech separation method, which combines Figure 1 , including nine steps S101 to S109:

[0050] S101: determining mixed training sets and test sets of people speaking under different space sizes and different microphone array types according to the in-vehicle space and M microphones by adopting the mirror source method;

[0051] Specifically, in the embodiment of the present application, it is assumed that M microphones are arranged in the vehicle, and the entire vehicle space is divided into four areas, each area having one microphone and at most one person moving in the area. According to the empirical risk minimization theory, the priori is added to the data to make the simulated data more approximate to the real data probability distribution. The mirror source method is used to determine the mixed training set and test set of 1 person, 2 persons, 3 persons, and 4 persons speaking under different space sizes and different microphone arrays according to the vehicle space and the M microphones, which respectively account for 50%, 20%, 20%, and 10%. The module 201 uses the mirror source method to simulate the noise in the real environment such as the background noise and wind noise. The sound source position of each space is generated according to Gaussian random sampling, and the room size and noise position are generated according to uniform random sampling.

[0052] S102: Convert the speech of the mixed training set to the frequency domain.

[0053] Specifically, in the embodiment of the present application, the mixed training set includes clean target speech and noise containing the clean target speech, and the method of converting the speech of the mixed training set to the frequency domain is described in detail below.

[0054] S103: Input the signal converted to the frequency domain into the network to obtain an enhanced speech signal.

[0055] Specifically, in the embodiment of the present application, the method of inputting the signal converted to the frequency domain into the network to obtain an enhanced speech signal is described in detail below.

[0056] S104: Calculate a loss function using the signal converted to the frequency domain and the enhanced speech signal.

[0057] Specifically, in the embodiment of the present application, the loss function is calculated using both the signal converted to the frequency domain and the enhanced speech signal.

[0058] S105: Update the network weight according to the loss function.

[0059] Specifically, in the embodiment of the present application, the method of updating the network weight according to the loss function is described in detail below.

[0060] S106: Train multiple rounds, process the test set in each round, and calculate the index of the test set.

[0061] Specifically, in the embodiment of the present application, the index of the test set can be SNR and SI-SNR.

[0062] S107: Select a model according to the index of the test set in multiple rounds.

[0063] Specifically, in the embodiment of the present application, one available model is selected according to the best index of the test set in multiple rounds.

[0064] S108: obtaining the frequency domain signals of the M channels through the model.

[0065] S109: changing the M-channel frequency domain speech signals to the time domain to obtain the final separated multi-channel speech signals.

[0066] Specifically, the method for changing the M-channel frequency domain speech signals to the time domain to obtain the final separated multi-channel speech signals is described in detail below.

[0067] Further, in the above-mentioned vehicle-mounted multi-channel speech separation method, the conversion of the mixed training set speech to the frequency domain includes:

[0068] frame division, Hanning window addition, and short-time Fourier conversion are performed on the mixed training set speech to convert the speech to the frequency domain.

[0069] In the above-mentioned vehicle-mounted multi-channel speech separation method, the conversion of the mixed training set speech to the frequency domain includes:

[0070] Further, in the above-mentioned vehicle-mounted multi-channel speech separation method, the conversion of the mixed training set speech to the frequency domain includes:

[0071] The converted signals are input into the network, and a frequency band division operation is performed on the frequency domain.

[0072] After the frequency band division operation, the signals are sequentially sent to the encoder, LSTM, and decoder layers.

[0073] The frequency band merging operation is performed to output the frequency domain signals of the M channels.

[0074] In the above-mentioned vehicle-mounted multi-channel speech separation method, the conversion of the mixed training set speech to the frequency domain includes:

[0075] Specifically, in the above-mentioned vehicle-mounted multi-channel speech separation method, the last layer of the decoder uses a full connection instead of the original transpose convolution.

[0076] Further, in the above-mentioned vehicle-mounted multi-channel speech separation method, the conversion of the mixed training set speech to the frequency domain includes:

[0077] In the above-mentioned vehicle-mounted multi-channel speech separation method, the conversion of the mixed training set speech to the frequency domain includes:

[0078] Specifically, in the above-mentioned vehicle-mounted multi-channel speech separation method, the conversion of the mixed training set speech to the frequency domain includes: all+βMSE i , where i∈(1,M).

[0079] Furthermore, in the above-mentioned in-vehicle multi-channel voice separation method, an asymmetric network is used.

[0080] Specifically, in this embodiment of the invention, an asymmetric network is used to reduce the computational load of the network.

[0081] Furthermore, in the above-mentioned in-vehicle multi-channel speech separation method, the M channel frequency domain speech signals are transformed to the time domain to obtain the final separated multi-channel speech signals, including:

[0082] The M-channel frequency domain speech signals are transformed to the time domain through inverse short-time Fourier transform, windowing, and overlapping summation to obtain the final separated multi-channel speech signals.

[0083] Specifically, in this embodiment of the invention, the output M-channel frequency domain speech signal is transformed to the time domain through inverse short-time Fourier transform, windowing, and overlapping addition to obtain the final separated multi-channel speech signal.

[0084] Furthermore, in the above-mentioned in-vehicle multi-channel speech separation method, the encoder layer adopts a grouped convolutional design.

[0085] Figure 2 This invention provides an in-vehicle multi-channel voice separation device.

[0086] Secondly, embodiments of the present invention also provide an in-vehicle multi-channel voice separation device, combined with Figure 2 ,include:

[0087] Determine module 201: Used to determine the mixed training set and test set for human speech under different spatial dimensions and different microphone arrays based on the in-vehicle space and M microphones using the mirror source method.

[0088] Specifically, in this embodiment of the invention, it is assumed that there are M microphones arranged inside the vehicle, and the entire vehicle space is divided into four areas. Each area contains one microphone and at most one person is active within that area. Based on the empirical risk minimization theory, prior information is added to the data to make the simulated data more closely approximate the real data probability distribution. The determination module 201 uses the mirror source method to determine mixed training and test sets for 1 person, 2 people, 3 people, and 4 people speaking under different space sizes and different microphone arrays, based on the vehicle space and the M microphones, with proportions of 50%, 20%, 20%, and 10%, respectively. The determination module 201 uses the mirror source method to simulate noise in the real environment, such as background noise and wind noise. The sound source position of each space is generated by Gaussian random sampling, and the room size and noise position are generated by uniform random sampling.

[0089] The conversion module 202 is configured to convert the speech in the mixed training set to the frequency domain.

[0090] Specifically, in the embodiment of the present application, the mixed training set includes clean target speech and noise containing the clean target speech, and the conversion module 202 converts the speech in the mixed training set to the frequency domain.

[0091] The training module 203 is configured to input the signal converted to the frequency domain into the network to obtain an enhanced speech signal after training.

[0092] Specifically, in the embodiment of the present application, the training module 203 inputs the signal converted to the frequency domain into the network to obtain an enhanced speech signal after training.

[0093] The first calculation module 204 is configured to calculate a loss function by using the signal converted to the frequency domain and the enhanced speech signal.

[0094] Specifically, in the embodiment of the present application, the first calculation module 204 calculates the loss function by using both the signal converted to the frequency domain and the enhanced speech signal.

[0095] The update module 205 is configured to update the network weight according to the loss function.

[0096] Specifically, in the embodiment of the present application, the update module 205 updates the network weight according to the loss function.

[0097] The second calculation module 206 is configured to train multiple rounds, process the test set in each round, and calculate an index of the test set.

[0098] Specifically, in the embodiment of the present application, the second calculation module 206 can calculate the SNR and the SI-SNR as the index of the test set.

[0099] The screening module 207 is configured to select a model according to the index of the test set in multiple rounds.

[0100] Specifically, in the embodiment of the present application, the screening module 207 selects a usable model according to the best index of the test set in multiple rounds.

[0101] The obtaining module 208 is configured to obtain the frequency domain signal of M channels through the model.

[0102] Specifically, in the embodiment of the present application, the obtaining module 208 obtains the frequency domain signal of M channels through the model.

[0103] The change module 209 is configured to change the output M-channel frequency domain speech signal to the time domain to obtain a final separated multi-channel speech signal.

[0104] Specifically, in this embodiment of the invention, the transformation module 209 transforms the M channel frequency domain speech signals to the time domain to obtain the final separated multi-channel speech signals.

[0105] The mixed training and test sets include: clean target speech and noise containing clean target speech.

[0106] Thirdly, embodiments of the present invention also provide an electronic device, including: a processor and a memory;

[0107] The processor executes an in-vehicle multi-channel voice separation method as described above by calling programs or instructions stored in the memory.

[0108] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a program or instructions that cause a computer to execute the in-vehicle multi-channel voice separation method described in any one of the above claims.

[0109] Figure 3 This is a schematic block diagram of an electronic device provided in an embodiment of this disclosure.

[0110] like Figure 3 As shown, the electronic device includes at least one processor 301, at least one memory 302, and at least one communication interface 303. The various components of the electronic device are coupled together via a bus system 304. The communication interface 303 is used for information transmission with external devices. It is understood that the bus system 304 is used to implement communication between these components. In addition to a data bus, the bus system 304 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 3 The general designated all buses as Bus System 304.

[0111] It is understood that the memory 302 in this embodiment may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0112] In some implementations, memory 302 stores elements such as executable units or data structures, or subsets thereof, or extended sets thereof: operating systems and applications.

[0113] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks. The application programs include various application programs, such as a media player (Media Player), a browser (Browser), and the like, for implementing various application services. The program for implementing any of the methods of the vehicle-mounted multi-channel speech separation method provided in the embodiments of the present application can be included in the application programs.

[0114] In the embodiments of the present application, the processor 301 invokes the program or instruction stored in the memory 302, specifically, the program or instruction stored in the application program, and the processor 301 is configured to execute the steps of each embodiment of the vehicle-mounted multi-channel speech separation method provided in the embodiments of the present application.

[0115] The mixed training set and the test set of the speech of the person speaking under different spatial sizes and different microphone array types are determined according to the in-vehicle space and the M microphones by using the mirror source method;

[0116] The speech of the mixed training set is converted to the frequency domain;

[0117] The signal converted to the frequency domain is input into the network to obtain the enhanced speech signal after training;

[0118] The loss function is calculated by using the signal converted to the frequency domain and the enhanced speech signal;

[0119] The network weight is updated according to the loss function;

[0120] Multiple rounds of training are performed, and in each round, the test set is processed, and the index of the test set is calculated;

[0121] The model is selected according to the index of the test set in the multiple rounds;

[0122] The frequency domain signal of the M channels is obtained by using the model;

[0123] The M-channel frequency domain speech signal is changed to the time domain to obtain the final separated multi-channel speech signal;

[0124] Any one of the vehicle-mounted multi-channel speech separation methods provided in the embodiments of the present application can be applied in the processor 301 or implemented by the processor 301. The processor 301 can be an integrated circuit chip with a signal processing capability. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 301 or an instruction in the form of software. The processor 501 described above can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general processor can be a microprocessor or the processor can also be any conventional processor.

[0125] The steps of any one of the vehicle-mounted multi-channel speech separation methods provided in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed and completed, or executed and completed by a combination of hardware and software units in the decoding processor. The software unit can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register or other mature storage media in the field. The storage media is located in the storage 302, and the processor 301 reads the information in the storage 302 and completes the steps of the method in combination with the hardware thereof.

[0126] Those skilled in the art can understand that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments means to be within the scope of the present application and form different embodiments.

[0127] Those skilled in the art can understand that the description of each embodiment is focused on, and the part not described in detail in a certain embodiment can refer to the related description of other embodiments.

[0128] Although the embodiments of the present application are described in conjunction with the drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes are intended to fall within the scope of the appended claims. Above, only specific embodiments of the present application are described, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0129] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements shall be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A vehicle-mounted multi-channel voice separation method, characterized in that, include: The mirror source method was used to determine mixed training and test sets for human speech under different space sizes and microphone arrays based on the vehicle interior space and M microphones. Convert the speech of the hybrid training set to the frequency domain; The signal converted to the frequency domain is input into the network and trained to obtain the enhanced speech signal. The loss function is calculated using the signal converted to the frequency domain and the enhanced speech signal. Update the network weights according to the loss function; The training process is repeated in multiple rounds, with the test set being processed in each round and the metrics for the test set being calculated. The model is selected based on the metrics of the test set in multiple rounds; The model yields frequency domain signals for M channels. The frequency domain signals of the M channels are transformed to the time domain to obtain the final separated multi-channel speech signal; The mixed training set and the test set both include: clean target speech and noise containing clean target speech.

2. The in-vehicle multi-channel voice separation method according to claim 1, characterized in that, Converting the speech of the hybrid training set to the frequency domain includes: The speech from the hybrid training set is converted to the frequency domain through frame segmentation, Hanning windowing, and short-time Fourier transform. The frame is 32ms per frame with a 50% frame overlap rate.

3. The in-vehicle multi-channel voice separation method according to claim 1, characterized in that, The enhanced speech signal, obtained by training the network after inputting the signal converted to the frequency domain, includes: The signal converted to the frequency domain is input into the network, and frequency band division is implemented in the frequency domain; After the frequency band division operation, the data is sequentially fed into the encoder, LSTM and decoder layers; Implement the band combining operation to output frequency domain signals of M channels; The last layer of the decoder layer is a fully connected layer.

4. The in-vehicle multi-channel voice separation method according to claim 1, characterized in that, Updating the network weights according to the loss function includes: The weights of the network are updated using a hybrid weighted loss function that combines the average mean square error of the M channels and the average mean square error between individual channels.

5. The in-vehicle multi-channel voice separation method according to claim 1, characterized in that, The network is an asymmetric network.

6. The in-vehicle multi-channel voice separation method according to claim 1, characterized in that, The frequency domain signals of the M channels are transformed to the time domain to obtain the final separated multi-channel speech signal, including: The frequency domain signals of M channels are transformed to the time domain through inverse short-time Fourier transform, windowing, and overlapping addition to obtain the final separated multi-channel speech signal.

7. The in-vehicle multi-channel voice separation method according to claim 3, characterized in that, The encoder layer employs a grouped convolutional design.

8. A vehicle-mounted multi-channel voice separation device, characterized in that, include: Determine the module: Used to determine the mixed training and test sets for human speech under different spatial dimensions and microphone arrays based on the in-vehicle space and M microphones using the mirror source method; Conversion module: used to convert the speech of the mixed training set to the frequency domain; Training module: Used to input the signal converted to the frequency domain into the network and train it to obtain the enhanced speech signal; First calculation module: used to calculate a loss function using the signal converted to the frequency domain and the enhanced speech signal; Update module: used to update network weights according to the loss function; The second calculation module is used for training multiple rounds, processing the test set in each round, and calculating the metrics of the test set. Filtering module: Used to select models based on metrics from the test set across multiple rounds; The obtaining module is used to obtain the frequency domain signals of M channels through the model. Transformation module: Used to transform the frequency domain signals of M channels to the time domain to obtain the final separated multi-channel speech signal; The mixed training set and the test set both include: clean target speech and noise containing clean target speech.

9. An electronic device, characterized in that, include: Processor and memory; The processor executes a vehicle-mounted multi-channel voice separation method as described in any one of claims 1 to 7 by calling the program or instructions stored in the memory.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that cause a computer to perform a vehicle-mounted multi-channel voice separation method as described in any one of claims 1 to 7.

Images