Voice processing method, device, equipment and storage medium

Abstract

The present disclosure provides a speech processing method, device, equipment and storage medium, and relates to the field of speech processing technology. In some embodiments of the present disclosure, an original sound signal is obtained; the original sound signal is estimated in orientation to obtain the positions of multiple sound sources contained in the original sound signal; multiple sound sources are beam separated to obtain initial sound signals of multiple sound sources; the voiceprint features of the target sound source in the original sound signal are extracted; the original sound signal, the positions of multiple sound sources, the initial sound signal and the voiceprint features of the target sound source are input into a voiceprint separation network to obtain the target sound signals of multiple sound sources; the present disclosure performs voiceprint separation of multiple sound sources based on the original sound signal, the positions of multiple sound sources, the initial sound signal and the voiceprint features of the target sound source, and the voiceprint separation network can improve the accuracy of voiceprint separation and improve generalization performance.

Description

Technical Field

[0001] The present disclosure relates to the field of speech processing technology, and in particular to a speech processing method, apparatus, device and storage medium. Background Art

[0002] Sound is one of the most natural, effective and convenient ways of communication between people and machines. However, due to the complexity and variability of the natural environment, sound source signals are often interfered by noise and non-target sound sources, which seriously affects the sound quality. How to achieve high-performance multi-source separation is a bottleneck problem faced by the new generation of real-time communication and offline communication equipment.

[0003] Voiceprint separation is an emerging speech separation technology that separates the sound source by obtaining the voiceprint features of the target speech in advance. However, the voiceprint separation performance is unstable and the generalization performance is poor. In addition, when the voice of the same-sex speaker or the target speaker changes due to a cold or other reasons, the separation performance is poor and even mis-separation may occur.

[0004] Therefore, the current voiceprint separation methods have low accuracy and poor generalization performance. Summary of the invention

[0005] The present disclosure provides a speech processing method, apparatus, device and storage medium to at least solve the problem that the existing voiceprint separation method has low accuracy and poor generalization performance.

[0006] The technical solution of the present disclosure is as follows:

[0007] The present disclosure provides a speech processing method, including:

[0008] Obtaining original sound signals;

[0009] Performing azimuth estimation on the original sound signal to obtain positions of multiple sound sources contained in the original sound signal;

[0010] Performing beam separation on the multiple sound sources to obtain initial sound signals of the multiple sound sources;

[0011] Extracting the voiceprint features of the target sound source in the original sound signal;

[0012] The original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source are input into a voiceprint separation network to obtain the target sound signals of the multiple sound sources.

[0013] Optionally, the performing azimuth estimation on the original sound signal to obtain positions of multiple sound sources contained in the original sound signal includes:

[0014] The original sound signal is estimated by using any one of the azimuth estimation algorithms, such as the traditional beamforming technology, the minimum variance distortion-free response technology, the multiple signal classification algorithm, and the compressed sensing technology, to obtain the positions of multiple sound sources contained in the original sound signal.

[0015] Optionally, performing beam separation on the multiple sound sources to obtain initial sound signals of the multiple sound sources includes:

[0016] Adaptive beamforming technology is used to perform adaptive beamforming on each sound source to obtain initial sound signals of multiple sound sources.

[0017] Optionally, extracting the voiceprint feature of the target sound source in the original sound signal includes:

[0018] Extracting voiceprint features of multiple sound sources;

[0019] Matching the voiceprint features of each of the sound sources with the sound features of the registered sound sources respectively;

[0020] A voiceprint feature of the target sound source that successfully matches the registered sound source is selected from the multiple sound sources.

[0021] Optionally, the voiceprint separation network includes: a convolutional neural network, a long short-term memory network, a fully connected layer and an output layer; the original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source are input into the voiceprint separation network to obtain the target sound signals of the multiple sound sources, including:

[0022] Inputting the original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source into a voiceprint separation network;

[0023] In the voiceprint separation network, the convolutional neural network is used to extract local image features of the time-frequency spectrum graph, and input the local image features into the long short-term memory network;

[0024] The long short-term memory network is used to process the local features of the image to obtain speech time series features, and input the speech time series features into the fully connected layer;

[0025] The fully connected layer is used to map the feature space corresponding to the speech time series feature to the sample label space to obtain the feature label value;

[0026] The output layer is used to convert the feature marker value into a time-frequency mask;

[0027] Speech separation is performed according to the time-frequency mask and the frequency domain data of the original sound signal to obtain target sound signals of multiple sound sources.

[0028] Optionally, performing speech separation according to the time-frequency mask and the frequency domain data of the original sound signal to obtain target sound signals of the plurality of sound sources includes:

[0029] The time-frequency mask and the frequency domain data of the original sound signal are point-multiplied to obtain a target sound signal of the sound source.

[0030] The present disclosure also provides a speech processing device, including:

[0031] An acquisition module, used for acquiring original sound signals;

[0032] A direction estimation module, used to estimate the direction of the original sound signal to obtain the positions of multiple sound sources contained in the original sound signal;

[0033] A separation module, used for performing beam separation on the multiple sound sources to obtain initial sound signals of the multiple sound sources;

[0034] An extraction module, used to extract the voiceprint features of the target sound source in the original sound signal;

[0035] The neural network module is used to input the original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source into a voiceprint separation network to obtain the target sound signals of the multiple sound sources.

[0036] Optionally, the direction estimation module, when performing direction estimation on the original sound signal to obtain positions of multiple sound sources contained in the original sound signal, is used to:

[0037] The original sound signal is estimated by using any one of the azimuth estimation algorithms, such as the traditional beamforming technology, the minimum variance distortion-free response technology, the multiple signal classification algorithm, and the compressed sensing technology, to obtain the positions of multiple sound sources contained in the original sound signal.

[0038] Optionally, when performing beam separation on the multiple sound sources to obtain the initial sound signals of the multiple sound sources, the separation module is used to:

[0039] Adaptive beamforming technology is used to perform adaptive beamforming on each sound source to obtain initial sound signals of multiple sound sources.

[0040] Optionally, when extracting the voiceprint feature of the target sound source in the original sound signal, the extraction module is used to:

[0041] Extracting voiceprint features of multiple sound sources;

[0042] Matching the voiceprint features of each of the sound sources with the sound features of the registered sound sources respectively;

[0043] A voiceprint feature of the target sound source that successfully matches the registered sound source is selected from the multiple sound sources.

[0044] Optionally, the voiceprint separation network includes: a convolutional neural network, a long short-term memory network, a fully connected layer and an output layer; when the original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source are input into the voiceprint separation network to obtain the target sound signals of the multiple sound sources, the neural network module is used to:

[0045] Inputting the original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source into a voiceprint separation network;

[0046] In the voiceprint separation network, the convolutional neural network is used to extract local image features of the time-frequency spectrum graph, and input the local image features into the long short-term memory network;

[0047] The long short-term memory network is used to process the local features of the image to obtain speech time series features, and input the speech time series features into the fully connected layer;

[0048] The fully connected layer is used to map the feature space corresponding to the speech time series feature to the sample label space to obtain the feature label value;

[0049] The output layer is used to convert the feature marker value into a time-frequency mask;

[0050] Speech separation is performed according to the time-frequency mask and the frequency domain data of the original sound signal to obtain target sound signals of multiple sound sources.

[0051] Optionally, when performing speech separation according to the time-frequency mask and the frequency domain data of the original sound signal to obtain the target sound signals of the plurality of sound sources, the neural network module is used to:

[0052] The time-frequency mask and the frequency domain data of the original sound signal are point-multiplied to obtain a target sound signal of the sound source.

[0053] The present disclosure also provides an electronic device, including:

[0054] processor;

[0055] a memory for storing processor-executable instructions;

[0056] The processor is configured to execute instructions to implement each step in the above method.

[0057] The embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each step in the above method is implemented.

[0058] The embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, which implements each step of the above method when executed by a processor.

[0059] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0060] In some embodiments of the present disclosure, an original sound signal is acquired; an azimuth estimation is performed on the original sound signal to obtain the positions of multiple sound sources contained in the original sound signal; beam separation is performed on the multiple sound sources to obtain initial sound signals of the multiple sound sources; voiceprint features of a target sound source in the original sound signal are extracted; the original sound signal, the positions of multiple sound sources, the initial sound signal, and the voiceprint features of the target sound source are input into a voiceprint separation network to obtain target sound signals of the multiple sound sources; the present disclosure performs voiceprint separation of multiple sound sources based on the original sound signal, the positions of multiple sound sources, the initial sound signal, and the voiceprint features of the target sound source, and the use of a voiceprint separation network can improve the accuracy of voiceprint separation and improve generalization performance.

[0061] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0063] Figure 1 A flowchart of a speech processing method provided for an exemplary embodiment of the present disclosure;

[0064] Figure 2 A structural schematic diagram of a speech separation system is provided for an exemplary embodiment of the present disclosure;

[0065] Figure 3 A schematic diagram of the structure of a speech processing device provided by an exemplary embodiment of the present disclosure;

[0066] Figure 4 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0067] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0068] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure.

[0069] It should be noted that the user information involved in this disclosure includes but is not limited to: user device information and user personal information; the collection, storage, use, processing, transmission, provision and disclosure of user information in this disclosure comply with the relevant laws and regulations and do not violate public order and good morals.

[0070] With the advancement of industrial equipment and the development of signal space sampling theory, microphone arrays are used to replace the original single microphone in car cockpits, mobile phones, speakers, TVs, tablets, computers, audio and video conferencing and other equipment or systems. Therefore, the problem of multi-channel multi-source separation based on microphone arrays has gradually become a research hotspot in academia and industry. Its research scope has expanded from the initial signal enhancement field to the cross-fields of array signal processing, acoustics, artificial intelligence, etc. How to achieve high-quality multi-source separation has always been the goal pursued by many research institutions and scholars in the world.

[0071] Voiceprint separation is an emerging speech separation technology that separates the sound source by obtaining the voiceprint features of the target speech in advance. However, the voiceprint separation performance is unstable and the generalization performance is poor. In addition, the separation performance is poor for the same-sex or target speakers with voice changes due to colds, and even mis-separation occurs. Therefore, the current voiceprint separation methods have low accuracy and poor generalization performance.

[0072] In response to the above technical problems, in some embodiments of the present disclosure, an original sound signal is obtained; the orientation of the original sound signal is estimated to obtain the positions of multiple sound sources contained in the original sound signal; beam separation is performed on the multiple sound sources to obtain initial sound signals of the multiple sound sources; the voiceprint features of the target sound source in the original sound signal are extracted; the original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source are input into a voiceprint separation network to obtain the target sound signal of the multiple sound sources; the present disclosure performs voiceprint separation of multiple sound sources based on the original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source, and the use of the voiceprint separation network can improve the accuracy of voiceprint separation and improve generalization performance.

[0073] The technical solutions provided by various embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.

[0074] Figure 1 The flowchart of a speech processing method provided by the exemplary embodiment of the present disclosure is as follows. Figure 1 As shown, the method includes:

[0075] S101: Acquire original sound signal;

[0076] S102: performing azimuth estimation on the original sound signal to obtain positions of multiple sound sources contained in the original sound signal;

[0077] S103: performing beam separation on multiple sound sources to obtain initial sound signals of the multiple sound sources;

[0078] S104: extracting the voiceprint features of the target sound source in the original sound signal;

[0079] S105: Inputting the original sound signal, the positions of multiple sound sources, the initial sound signal and the voiceprint features of the target sound source into a voiceprint separation network to obtain the target sound signals of the multiple sound sources.

[0080] In the embodiments of the present disclosure, the execution subject of the above method may be a terminal device and a server.

[0081] Among them, the terminal device includes but is not limited to a mobile station (MS), a mobile terminal, a mobile phone, a handset and a portable equipment, etc. The terminal device can communicate with one or more core networks via a radio access network (RAN). For example, the terminal device can be a mobile phone (or a "cellular" phone), a computer with a wireless communication function, etc. The terminal device can also be a computer with a wireless transceiver function, a virtual reality (VR) terminal device, an AR terminal device, a wireless terminal in industrial control, a wireless terminal in self driving, a wireless terminal in remote medical, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc., and the operating system installed on the terminal device includes but is not limited to: IOS, Android, windows, linux, Mac OS and other operating systems. In different networks, the terminal may be called different names, such as user equipment, mobile station, subscriber unit, station, cellular phone, personal digital assistant, wireless modem, wireless communication device, handheld device, laptop computer, cordless phone, wireless local loop station, television, etc. For the convenience of description, it is referred to as terminal equipment in this embodiment.

[0082] In this embodiment, the implementation form of the server is not limited. For example, the server can be a conventional server, a cloud server, a cloud host, a virtual center, or other server devices. The server mainly includes a processor, a hard disk, a memory, a system bus, etc., and a general computer architecture type.

[0083] In this embodiment, an original sound signal is obtained; the orientation of the original sound signal is estimated to obtain the positions of multiple sound sources contained in the original sound signal; beam separation is performed on the multiple sound sources to obtain initial sound signals of the multiple sound sources; the voiceprint features of the target sound source in the original sound signal are extracted; the original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source are input into a voiceprint separation network to obtain the target sound signals of the multiple sound sources; the present invention performs voiceprint separation of multiple sound sources based on the original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source, and the use of the voiceprint separation network can improve the accuracy of voiceprint separation and improve generalization performance.

[0084] In the disclosed embodiment, a method combining traditional microphone array speech processing technology and voiceprint separation network is used to improve the stability and performance of voiceprint separation. First, the original sound signal collected by the microphone array is processed, and the original sound signal is estimated to obtain the positions of multiple sound sources contained in the original sound signal; secondly, multiple sound sources are beam separated to obtain the initial sound signals of multiple sound sources; the voiceprint features of the target sound source in the original sound signal are extracted; finally, the above original sound signal, the positions of multiple sound sources, the initial sound signal and the voiceprint features of the target sound source are input into the voiceprint separation network to perform sound source separation to obtain the output signal of each sound source.

[0085] It should be noted that the voice processing method disclosed in the present invention can be applied to technical fields such as mobile phones and car cockpits. The voice processing method disclosed in the present invention can be used for multi-zone calls, and can also be used for interference suppression calls. Further application scenarios are voice wake-up and voice assistant scenarios.

[0086] In some embodiments of the present disclosure, original sound signals are acquired, for example, by using a microphone array to acquire original sound signals in an environment.

[0087] In an optional embodiment, the original sound signal is a time domain signal, and the time domain signal can be converted into other domain signals according to actual conditions. For example, if a frequency domain signal is required in a subsequent processing step, the original sound signal can be converted into a frequency domain signal through Fourier transform.

[0088] In some embodiments of the present disclosure, since the sound sources are often distributed in different positions in space and therefore have different angles, it is necessary to use an azimuth estimation technique to perform azimuth estimation on the sound source signal. The original sound signal is estimated in azimuth to obtain the positions of multiple sound sources contained in the original sound signal. The original sound signal is estimated in azimuth using any one of the azimuth estimation algorithms, including conventional beamforming (CBF), minimum variance distortionless response (MVDR), multiple signal classification (MUSIC), and compressed sensing (CS), to obtain the positions of multiple sound sources contained in the original sound signal.

[0089] In some embodiments of the present disclosure, multiple sound sources are beam separated to obtain initial sound signals of the multiple sound sources. One achievable method is to use adaptive beamforming technology to perform adaptive beamforming on each sound source to obtain initial sound signals of multiple sound sources. The present disclosure uses adaptive beamforming technology, i.e. low sidelobe adaptive beamforming or conventional adaptive beamforming technology, to suppress interference from other sound sources. When multiple sound sources are detected, corresponding adaptive beamforming is performed on each sound source. For example, for a car cockpit, if the driver and the co-driver speak at the same time, two beams need to be designed. When the main lobe of the beam points to the driver, the signal from the co-driver's position is suppressed, thereby extracting the driver's voice signal; when the main lobe of the beam points to the co-driver, the signal from the driver's position is suppressed, thereby extracting the co-driver's voice signal. These two extraction processes can be processed simultaneously through multi-threading to improve the efficiency of sound source separation.

[0090] In some embodiments of the present disclosure, the voiceprint features of the target sound source in the original sound signal are extracted. One achievable method is to extract the voiceprint features of multiple sound sources; match the voiceprint features of each sound source with the sound features of the registered sound source respectively; and select the voiceprint features of the target sound source that successfully matches the registered sound source from the multiple sound sources. Exemplarily, it is necessary to register the voiceprint of the target sound source in advance, and the audio of the target sound source can be pre-recorded for subsequent voiceprint feature extraction, which is equivalent to obtaining the characteristic information such as the sound quality of the target sound source in advance, which is a very strong prior knowledge. It should be noted that the present disclosure does not limit the number of target sound sources, and the number of target sound sources can be one or more. The matching process of multiple sound sources and the registered sound source can be matched using a similarity algorithm. When the similarity condition is met, the target sound source and the registered sound source are successfully matched.

[0091] In an exemplary embodiment, the audio of the main driver and the co-driver are pre-recorded in the car cabin as the voiceprint of the target sound source; after the initial sound information of the three sound sources is identified, the initial sound information of each sound source is respectively calculated for similarity with the audio of the main driver and the co-driver, and after the initial sound signals of two of the three sound sources are successfully matched with the audio of the main driver and the co-driver, the voiceprint features of the initial sound signals of the two successfully matched sound sources are extracted. The present disclosure can improve the accuracy of the final voiceprint separation by pre-recording the voiceprint of the target sound source.

[0092] In some embodiments of the present disclosure, the original sound signal, the positions of multiple sound sources, the initial sound signal and the voiceprint features of the target sound source are fused, for example, the information is directly spliced, etc. It should be noted that there is no restriction on the fusion method, and then the fused information is sent to the voiceprint separation network.

[0093] Figure 2 The present invention provides a schematic diagram of a speech separation system according to an exemplary embodiment of the present invention. Figure 2 As shown, the voiceprint separation network includes: Convolutional Neural Networks (CNN), Long Short-term Memory (LSTM), Fully Connect and output layer. It should be noted that the present disclosure does not limit the number of layers and convolution kernels of the convolutional neural network. The present disclosure does not limit the number of layers of the long short-term memory network. Figure 2 There are four convolutional neural networks and three long short-term memory networks.

[0094] In some embodiments of the present disclosure, the original sound signal, the positions of multiple sound sources, the initial sound signal, and the voiceprint features of the target sound source are input into a voiceprint separation network to obtain the target sound signals of the multiple sound sources. One achievable method is to input the original sound signal, the positions of multiple sound sources, the initial sound signal, and the voiceprint features of the target sound source into the voiceprint separation network; in the voiceprint separation network, the convolutional neural network is used to extract the image local features of the time-frequency spectrum, and the image local features are input into the long short-term memory network; the long short-term memory network is used to process the image local features to obtain the speech time series features, and the speech time series features are input into the fully connected layer; the fully connected layer is used to map the feature space corresponding to the speech time series features to the sample label space to obtain the feature label value; the output layer is used to convert the feature label value into a time-frequency mask; speech separation is performed according to the time-frequency mask and the frequency domain data of the original sound signal to obtain the target sound signals of the multiple sound sources. The present invention performs voiceprint separation of multiple sound sources based on the original sound signal, the positions of multiple sound sources, the initial sound signal and the voiceprint features of the target sound source. The voiceprint separation network is used to improve the accuracy of voiceprint separation and the generalization performance.

[0095] In the above embodiments, convolutional neural networks were first applied in the field of visual image processing. This is because convolutional neural networks can effectively perceive the local features of images. Since the time-frequency spectrum of the short-time Fourier transform of the speech signal belongs to a two-dimensional image and has an obvious harmonic structure, the harmonics are sparse in the entire spectrum range, and the harmonic structure belongs to the local features of the image. Therefore, convolutional neural networks have also been successfully applied in the field of speech signal processing. In a convolutional neural network, the size of the convolution kernel and the number of hidden layers of the neural network jointly determine the final receptive field of each convolution kernel.

[0096] In the above embodiment, since the speech signal is a time series, the convolutional neural network can only perceive local feature information and does not take into account the order between patterns. The long short-term memory network has the characteristic of node self-circulation, and the output of each node is related to the information of the current node and the historical node. This feature enables the recurrent neural network to be used to model sequence signals such as speech.

[0097] In the above embodiment, the main function of the fully connected layer is to map the feature space calculated by the previous layer to the sample label space, that is, to integrate the feature representation into a feature label value, which has the advantage of reducing the influence of the feature position on the classification result and improving the robustness of the entire network.

[0098] In some embodiments of the present disclosure, the output layer is used to convert the feature marker value into a time-frequency mask, that is, the time-frequency mask of the speech separation signal. The output layer can use a sigmoid function to convert the feature marker value into a time-frequency mask, and the calculation formula is as follows:

[0099]

[0100] Among them, x represents the feature marker value, f(x) represents the time-frequency mask; the time-frequency mask is generally a value between 0 and 1 or -1 and 1, and the separation of the sound source signal is achieved through the time-frequency mask.

[0101] In some embodiments of the present disclosure, speech separation is performed based on the time-frequency mask and the frequency domain data of the original sound signal to obtain target sound signals of multiple sound sources. One achievable method is to perform a dot multiplication of the time-frequency mask and the frequency domain data of the original sound signal to obtain the target sound signal of each sound source. Among them, the frequency domain data of the original sound signal can be converted into frequency domain data by Fourier transform.

[0102] In the above method embodiment of the present disclosure, an original sound signal is obtained; the orientation of the original sound signal is estimated to obtain the positions of multiple sound sources contained in the original sound signal; beam separation is performed on the multiple sound sources to obtain initial sound signals of the multiple sound sources; the voiceprint features of the target sound source in the original sound signal are extracted; the original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source are input into a voiceprint separation network to obtain the target sound signals of the multiple sound sources; the present disclosure performs voiceprint separation of multiple sound sources based on the original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source, and the use of the voiceprint separation network can improve the accuracy of voiceprint separation and improve generalization performance.

[0103] Figure 3 FIG. 3 is a schematic diagram of a structure of a speech processing device 30 provided by an exemplary embodiment of the present disclosure. Figure 3 As shown, the speech processing device 30 includes: an acquisition module 31, a direction estimation module 32, a separation module 33, an extraction module 34 and a neural network module 35.

[0104] Wherein, the acquisition module 31 is used to acquire the original sound signal;

[0105] A direction estimation module 32, used to estimate the direction of the original sound signal to obtain the positions of multiple sound sources contained in the original sound signal;

[0106] A separation module 33, used to perform beam separation on multiple sound sources to obtain initial sound signals of the multiple sound sources;

[0107] An extraction module 34, used to extract the voiceprint features of the target sound source in the original sound signal;

[0108] The neural network module 35 is used to input the original sound signal, the positions of multiple sound sources, the initial sound signal and the voiceprint features of the target sound source into the voiceprint separation network to obtain the target sound signal of the multiple sound sources.

[0109] Optionally, the direction estimation module 32, when performing direction estimation on the original sound signal to obtain the positions of multiple sound sources contained in the original sound signal, is used to:

[0110] The original sound signal is estimated by using any one of the azimuth estimation algorithms, such as the traditional beamforming technology, the minimum variance distortion-free response technology, the multiple signal classification algorithm, and the compressed sensing technology, to obtain the positions of multiple sound sources contained in the original sound signal.

[0111] Optionally, when the separation module 33 performs beam separation on multiple sound sources to obtain initial sound signals of the multiple sound sources, it is used to:

[0112] Adaptive beamforming technology is used to perform adaptive beamforming on each sound source to obtain initial sound signals of multiple sound sources.

[0113] Optionally, when extracting the voiceprint feature of the target sound source in the original sound signal, the extraction module 34 is used to:

[0114] Extract voiceprint features of multiple sound sources;

[0115] Match the voiceprint features of each sound source with the sound features of the registered sound source;

[0116] The voiceprint feature of the target sound source that successfully matches the registered sound source is selected from multiple sound sources.

[0117] Optionally, the voiceprint separation network includes: a convolutional neural network, a long short-term memory network, a fully connected layer and an output layer; when the neural network module 35 inputs the original sound signal, the positions of multiple sound sources, the initial sound signal and the voiceprint features of the target sound source into the voiceprint separation network to obtain the target sound signals of the multiple sound sources, it is used to:

[0118] Inputting the original sound signal, the positions of multiple sound sources, the initial sound signal and the voiceprint features of the target sound source into the voiceprint separation network;

[0119] In the voiceprint separation network, the convolutional neural network is used to extract the local image features of the time-frequency spectrum graph, and the local image features are input into the long short-term memory network;

[0120] The long short-term memory network is used to process the local features of the image to obtain the speech time sequence features, and the speech time sequence features are input into the fully connected layer;

[0121] The fully connected layer is used to map the feature space corresponding to the speech time series features to the sample tag space to obtain the feature tag value;

[0122] The output layer is used to convert feature marker values ​​into time-frequency masks;

[0123] Speech separation is performed based on the time-frequency mask and the frequency domain data of the original sound signal to obtain target sound signals of multiple sound sources.

[0124] Optionally, when performing speech separation according to the time-frequency mask and the frequency domain data of the original sound signal to obtain target sound signals of multiple sound sources, the neural network module 35 is used to:

[0125] The time-frequency mask is multiplied by the frequency domain data of the original sound signal to obtain the target sound signal of the sound source.

[0126] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0127] Figure 4 FIG. 1 is a schematic diagram of a structure of an electronic device provided by an exemplary embodiment of the present disclosure. Figure 4 As shown, the electronic device includes: a memory 41 and a processor 42. In addition, the electronic device also includes a power supply component 43 and a communication component 44.

[0128] The memory 41 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application program or method operating on the electronic device.

[0129] The memory 41 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0130] The communication component 44 is used for data transmission with other devices.

[0131] The processor 42 can execute computer instructions stored in the memory 41 to: obtain an original sound signal; estimate the direction of the original sound signal to obtain the positions of multiple sound sources contained in the original sound signal; perform beam separation on the multiple sound sources to obtain initial sound signals of the multiple sound sources; extract the voiceprint features of the target sound source in the original sound signal; input the original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source into a voiceprint separation network to obtain target sound signals of the multiple sound sources.

[0132] Optionally, when the processor 42 estimates the direction of the original sound signal to obtain the positions of multiple sound sources contained in the original sound signal, it is configured to:

[0133] The original sound signal is estimated by using any one of the azimuth estimation algorithms, such as the traditional beamforming technology, the minimum variance distortion-free response technology, the multiple signal classification algorithm, and the compressed sensing technology, to obtain the positions of multiple sound sources contained in the original sound signal.

[0134] Optionally, when the processor 42 performs beam separation on multiple sound sources to obtain initial sound signals of the multiple sound sources, it is configured to:

[0135] Adaptive beamforming technology is used to perform adaptive beamforming on each sound source to obtain initial sound signals of multiple sound sources.

[0136] Optionally, when extracting the voiceprint feature of the target sound source in the original sound signal, the processor 42 is used to:

[0137] Extract voiceprint features of multiple sound sources;

[0138] Match the voiceprint features of each sound source with the sound features of the registered sound source;

[0139] The voiceprint feature of the target sound source that successfully matches the registered sound source is selected from multiple sound sources.

[0140] Optionally, the voiceprint separation network includes: a convolutional neural network, a long short-term memory network, a fully connected layer and an output layer; when the processor 42 inputs the original sound signal, the positions of multiple sound sources, the initial sound signal and the voiceprint features of the target sound source into the voiceprint separation network to obtain the target sound signals of the multiple sound sources, it is used to:

[0141] Inputting the original sound signal, the positions of multiple sound sources, the initial sound signal and the voiceprint features of the target sound source into the voiceprint separation network;

[0142] In the voiceprint separation network, the convolutional neural network is used to extract the local image features of the time-frequency spectrum graph, and the local image features are input into the long short-term memory network;

[0143] The long short-term memory network is used to process the local features of the image to obtain the speech time sequence features, and the speech time sequence features are input into the fully connected layer;

[0144] The fully connected layer is used to map the feature space corresponding to the speech time series features to the sample tag space to obtain the feature tag value;

[0145] The output layer is used to convert feature marker values ​​into time-frequency masks;

[0146] Speech separation is performed based on the time-frequency mask and the frequency domain data of the original sound signal to obtain target sound signals of multiple sound sources.

[0147] Optionally, when the processor 42 performs speech separation according to the time-frequency mask and the frequency domain data of the original sound signal to obtain the target sound signals of multiple sound sources, it is configured to:

[0148] The time-frequency mask is multiplied by the frequency domain data of the original sound signal to obtain the target sound signal of the sound source.

[0149] Accordingly, the present disclosure also provides a computer-readable storage medium storing a computer program. When the computer-readable storage medium stores the computer program and the computer program is executed by one or more processors, the one or more processors execute Figure 1 Each step in the method embodiment.

[0150] Accordingly, the present disclosure also provides a computer program product, the computer program product includes a computer program / instruction, the computer program / instruction is executed by a processor Figure 1 Each step in the method embodiment.

[0151] Above Figure 4 The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0152] Above Figure 4 The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0153] The electronic device also includes a display screen and an audio component.

[0154] The display screen includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0155] The audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0156] In the above-mentioned apparatus, device, storage medium and computer program product embodiments of the present disclosure, an original sound signal is obtained; the original sound signal is estimated in orientation to obtain the positions of multiple sound sources contained in the original sound signal; beam separation is performed on the multiple sound sources to obtain initial sound signals of the multiple sound sources; the voiceprint features of the target sound source in the original sound signal are extracted; the original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source are input into a voiceprint separation network to obtain the target sound signals of the multiple sound sources; the present disclosure performs voiceprint separation of multiple sound sources based on the original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source, and the use of the voiceprint separation network can improve the accuracy of voiceprint separation and improve generalization performance.

[0157] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0158] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0159] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0161] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0162] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0163] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0164] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0165] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech processing method, characterized in that: include: Obtaining original sound signals; Performing azimuth estimation on the original sound signal to obtain positions of multiple sound sources contained in the original sound signal; Performing beam separation on the multiple sound sources to obtain initial sound signals of the multiple sound sources; Extracting the voiceprint features of the target sound source in the original sound signal; Inputting the original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source into a voiceprint separation network, wherein the voiceprint separation network includes: a convolutional neural network, a long short-term memory network, a fully connected layer and an output layer; In the voiceprint separation network, the convolutional neural network is used to extract local image features of the time-frequency spectrum graph, and input the local image features into the long short-term memory network; The long short-term memory network is used to process the local features of the image to obtain speech time series features, and input the speech time series features into the fully connected layer; The fully connected layer is used to map the feature space corresponding to the speech time series feature to the sample label space to obtain the feature label value; The output layer is used to convert the feature marker value into a time-frequency mask; Speech separation is performed according to the time-frequency mask and the frequency domain data of the original sound signal to obtain target sound signals of multiple sound sources.

2. The method according to claim 1, characterized in that The step of estimating the position of the original sound signal to obtain positions of multiple sound sources contained in the original sound signal includes: The original sound signal is estimated by using any one of the azimuth estimation algorithms, such as the traditional beamforming technology, the minimum variance distortion-free response technology, the multiple signal classification algorithm, and the compressed sensing technology, to obtain the positions of multiple sound sources contained in the original sound signal.

3. The method according to claim 1, characterized in that The step of performing beam separation on the plurality of sound sources to obtain initial sound signals of the plurality of sound sources comprises: Adaptive beamforming technology is used to perform adaptive beamforming on each sound source to obtain initial sound signals of multiple sound sources.

4. The method according to claim 1, characterized in that The extracting the voiceprint feature of the target sound source in the original sound signal includes: Extracting voiceprint features of multiple sound sources; Matching the voiceprint features of each of the sound sources with the sound features of the registered sound sources respectively; A voiceprint feature of the target sound source that successfully matches the registered sound source is selected from the multiple sound sources.

5. The method according to claim 1, characterized in that The step of performing speech separation according to the time-frequency mask and the frequency domain data of the original sound signal to obtain target sound signals of the plurality of sound sources comprises: The time-frequency mask and the frequency domain data of the original sound signal are point-multiplied to obtain a target sound signal of the sound source.

6. A speech processing device, characterized in that: include: An acquisition module, used for acquiring original sound signals; A direction estimation module, used to estimate the direction of the original sound signal to obtain the positions of multiple sound sources contained in the original sound signal; A separation module, used for performing beam separation on the multiple sound sources to obtain initial sound signals of the multiple sound sources; An extraction module, used to extract the voiceprint features of the target sound source in the original sound signal; A neural network module, configured to input the original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source into a voiceprint separation network to obtain the target sound signals of the multiple sound sources; The voiceprint separation network includes: a convolutional neural network, a long short-term memory network, a fully connected layer and an output layer; when the original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source are input into the voiceprint separation network to obtain the target sound signals of the multiple sound sources, the neural network module is used to: Inputting the original sound signal, the positions of the multiple sound sources, the initial sound signal and the voiceprint features of the target sound source into a voiceprint separation network; In the voiceprint separation network, the convolutional neural network is used to extract local image features of the time-frequency spectrum graph, and input the local image features into the long short-term memory network; The long short-term memory network is used to process the local features of the image to obtain speech time series features, and input the speech time series features into the fully connected layer; The fully connected layer is used to map the feature space corresponding to the speech time series feature to the sample label space to obtain the feature label value; The output layer is used to convert the feature marker value into a time-frequency mask; Speech separation is performed according to the time-frequency mask and the frequency domain data of the original sound signal to obtain target sound signals of multiple sound sources.

7. The device according to claim 6, characterized in that The direction estimation module, when performing direction estimation on the original sound signal to obtain the positions of multiple sound sources contained in the original sound signal, is used to: The original sound signal is estimated by using any one of the azimuth estimation algorithms, such as the traditional beamforming technology, the minimum variance distortion-free response technology, the multiple signal classification algorithm, and the compressed sensing technology, to obtain the positions of multiple sound sources contained in the original sound signal.

8. The device according to claim 6, characterized in that When the separation module performs beam separation on the multiple sound sources to obtain the initial sound signals of the multiple sound sources, it is used to: Adaptive beamforming technology is used to perform adaptive beamforming on each sound source to obtain initial sound signals of multiple sound sources.

9. The device according to claim 6, characterized in that When extracting the voiceprint feature of the target sound source in the original sound signal, the extraction module is used to: Extracting voiceprint features of multiple sound sources; Matching the voiceprint features of each of the sound sources with the sound features of the registered sound sources respectively; A voiceprint feature of the target sound source that successfully matches the registered sound source is selected from the multiple sound sources.

10. The device according to claim 6, characterized in that When the neural network module performs speech separation according to the time-frequency mask and the frequency domain data of the original sound signal to obtain the target sound signals of the plurality of sound sources, it is used to: The time-frequency mask and the frequency domain data of the original sound signal are point-multiplied to obtain a target sound signal of the sound source.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute instructions to implement each step in the method according to any one of claims 1 to 5.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the method according to any one of claims 1 to 5 is implemented.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images