Voice processing method, device, electronic device and storage medium

Abstract

The present disclosure proposes a speech processing method, device, electronic device and storage medium, which relate to the field of speech processing technology. The method comprises: obtaining a mixed speech, wherein the mixed speech includes a target speech corresponding to a target speaker and an interfering speech corresponding to an interfering speaker; extracting a first speech corresponding to a first direction where the target speaker is located and a second speech corresponding to a second direction where the interfering speaker is located from the mixed speech; determining a first spectrum corresponding to the first speech, a second spectrum corresponding to the second speech, and a third spectrum corresponding to the mixed speech; and separating the target speech corresponding to the target speaker from the mixed speech according to the first spectrum, the second spectrum and the third spectrum. In this way, the target speech corresponding to the target speaker can be accurately separated from the mixed signal, thereby improving the quality of the separated target speech.

Description

Technical Field

[0001] The present disclosure relates to the field of speech processing technology, and in particular to a speech processing method, device, electronic device and storage medium. Background Art

[0002] The cocktail party problem is a traditional problem in the field of speech signal processing. For example, in a public place, there may be multiple speakers speaking at the same time, and there may also be various environmental noises, background music, etc. This is a typical cocktail party problem. Therefore, when a microphone array is used to simultaneously receive mixed speech signals from multiple different sound sources, how to separate the speech signal of the target speaker from the mixed speech signal is an urgent problem to be solved. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a speech processing method, device, electronic device and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided a speech processing method, comprising:

[0005] Acquire a mixed speech, wherein the mixed speech includes a target speech corresponding to a target speaker and an interfering speech corresponding to an interfering speaker;

[0006] Extracting from the mixed speech a first speech corresponding to a first direction where the target speaker is located and a second speech corresponding to a second direction where the interfering speaker is located;

[0007] Determine a first spectrum corresponding to the first speech, a second spectrum corresponding to the second speech, and a third spectrum corresponding to the mixed speech;

[0008] The target speech corresponding to the target speaker is separated from the mixed speech according to the first spectrum, the second spectrum and the third spectrum.

[0009] According to a second aspect of an embodiment of the present disclosure, there is provided a speech processing device, including:

[0010] An acquisition module, configured to acquire a mixed speech, wherein the mixed speech includes a target speech corresponding to a target speaker and an interfering speech corresponding to an interfering speaker;

[0011] An extraction module, configured to extract, from the mixed speech, a first speech corresponding to a first direction where the target speaker is located and a second speech corresponding to a second direction where the interfering speaker is located;

[0012] A determination module, used to determine a first spectrum corresponding to the first speech, a second spectrum corresponding to the second speech, and a third spectrum corresponding to the mixed speech;

[0013] The separation module is used to separate the target speech corresponding to the target speaker from the mixed speech according to the first spectrum, the second spectrum and the third spectrum.

[0014] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to: implement the steps of the speech processing method proposed in the embodiment of the first aspect of the present disclosure.

[0015] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of a mobile terminal, the steps of the speech processing method proposed in the embodiment of the first aspect of the present disclosure are implemented.

[0016] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0017] In the disclosed embodiment, a first speech corresponding to the first direction where the target speaker is located and a second speech corresponding to the second direction where the interfering speaker is located are extracted from the mixed speech, and then a first spectrum corresponding to the first speech, a second spectrum corresponding to the second speech, and a third spectrum corresponding to the mixed speech are determined, and finally, the target speech corresponding to the target speaker is separated from the mixed speech according to the first spectrum, the second spectrum, and the third spectrum. Thus, the target speech corresponding to the target speaker can be accurately separated from the mixed signal by combining the first spectrum corresponding to the first speech in the first direction where the target speaker is located, the second spectrum corresponding to the second speech in the second direction where the interfering speaker is located, and the third spectrum corresponding to the mixed speech, thereby improving the quality of the separated target speech.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure:

[0020] Figure 1 is a flowchart of a speech processing method according to some embodiments of the present disclosure;

[0021] Figure 2 A schematic diagram of a process for obtaining a target speech corresponding to a target speaker provided by an embodiment of the present disclosure;

[0022] Figure 3 is a flowchart of a speech processing method according to some embodiments of the present disclosure;

[0023] Figure 4 A schematic diagram of a multi-zone communication system provided by an embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of another multi-zone communication provided by an embodiment of the present disclosure;

[0025] Figure 6 A schematic diagram of another multi-zone communication provided by an embodiment of the present disclosure;

[0026] Figure 7 A schematic diagram of a display interface provided by an embodiment of the present disclosure;

[0027] Figure 8 is a structural schematic diagram of a speech processing device according to some embodiments of the present disclosure;

[0028] Fig. 9 It is a schematic diagram of a functional block diagram of a vehicle showing an exemplary embodiment. DETAILED DESCRIPTION

[0029] Some embodiments of the present disclosure will be described in detail here, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. Various changes, modifications and equivalents of the methods, devices and / or systems described herein will become apparent after understanding the present disclosure. For example, the order of operations described herein is merely an example and is not limited to those orders set forth herein, but can be changed as becomes apparent after understanding the present disclosure, except for operations that must be performed in a specific order. In addition, for clarity and brevity, descriptions of features known in the art may be omitted.

[0030] The embodiments described in some embodiments of the present disclosure below do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0031] Figure 1 is a flow chart of a speech processing method according to some embodiments of the present disclosure. Figure 1 As shown, the voice processing method is used in a terminal, comprising the following steps:

[0032] Step 101: Acquire mixed speech, wherein the mixed speech includes a target speech corresponding to a target speaker and an interfering speech corresponding to an interfering speaker.

[0033] In some embodiments, the mixed speech may be speech of multiple speakers in the surrounding area collected by the microphone array. The target speaker may be one or more of the multiple speakers, and the interfering speaker may be other speakers except the target speaker.

[0034] In some embodiments, the target speaker may also be determined in advance, and then when it is detected that the microphone array collects speech including the target speaker and interfering speakers other than the target speaker, the collected speech is used as the target speaker.

[0035] Step 102: extracting from the mixed speech a first speech corresponding to a first direction where the target speaker is located and a second speech corresponding to a second direction where the interfering speaker is located.

[0036] In some embodiments, the sound source of the mixed speech is localized to obtain a first direction and a second direction, and the main lobe of the beam formed by the microphone array is directed to the first direction, and the side lobe of the beam is directed to the second direction, so as to extract the first speech from the mixed speech, wherein the microphone array is used to collect the mixed speech, and the main lobe of the beam formed by the microphone array is directed to the second direction, and the side lobe of the beam is directed to the first direction, so as to extract the second speech from the mixed speech.

[0037] The first speech contains a large amount of target speaker speech and a small amount of interfering speaker speech.

[0038] The second speech contains a large amount of interfering speaker speech and a small amount of target speaker speech.

[0039] It should be noted that by pointing the main lobe of the beam in the first direction and the side lobe of the beam in the second direction, the voice of the interfering speaker can be suppressed, so as to obtain the first voice in the first direction where the target speaker is located. By pointing the main lobe of the beam in the second direction and the side lobe of the beam in the first direction, the voice of the target speaker can be suppressed to obtain the second voice corresponding to the second direction where the interfering speaker is located.

[0040] It should be noted that since the beamforming technology cannot completely suppress the noise outside the beam direction, and noise will be mixed into the main lobe of the beam, there will be residual noise in the extracted first speech and the second speech. Post-processing filtering technology is required to continue filtering the extracted first speech and the second speech to improve the signal-to-noise ratio and thereby improve the clarity of the call.

[0041] It should be noted that, in some embodiments, the mixed signal may be replicated so as to extract the first voice and the second voice in multiple threads, thereby improving the efficiency of obtaining the first voice and the second voice.

[0042] Step 103: determine a first spectrum corresponding to the first speech, a second spectrum corresponding to the second speech, and a third spectrum corresponding to the mixed speech.

[0043] In some embodiments, Fourier transform may be performed on the first speech, the second speech, and the third speech respectively to obtain a first spectrum corresponding to the first speech, a second spectrum corresponding to the second speech, and a third spectrum corresponding to the mixed speech.

[0044] Step 104 : Separate the target speech corresponding to the target speaker from the mixed speech according to the first spectrum, the second spectrum and the third spectrum.

[0045] In some embodiments, the first spectrum, the second spectrum, and the third spectrum are input into a speech separation model to obtain a time-frequency mask corresponding to the target speaker, and then the time-frequency mask is multiplied by the third spectrum to obtain a fourth spectrum corresponding to the target speech, and finally, based on the fourth spectrum, the target speech corresponding to the target speaker is determined.

[0046] The speech separation model may be generated in advance based on training of a sample data set. The sample data set may include a first sample speech corresponding to a pure sample target speaker, a second sample speech corresponding to a pure sample interference speaker, and a mixed sample speech obtained by fusing the first sample speech and the second sample speech. The training process may include: Fourier transforming the first sample speech, the second sample speech, and the mixed sample speech to obtain a first sample spectrum corresponding to the first sample speech, a second sample spectrum corresponding to the second sample speech, and a third sample spectrum corresponding to the third sample speech, inputting the first sample spectrum, the second sample spectrum, and the third sample spectrum into the initial speech separation model to obtain a predicted time-frequency mask output by the initial speech separation model, and modifying the initial speech separation model according to the difference between the predicted time-frequency mask and the first sample spectrum to obtain a speech separation model.

[0047] In some embodiments, the speech separation model includes Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM) and a fully connected network.

[0048] Among them, the convolutional neural network can realize weight sharing, perceive the local features of the first spectrum, the second spectrum, and the third spectrum, realize the translation invariance of the two-dimensional features, and ensure that the network is robust to the two-dimensional features after rotation and translation. Therefore, the impact of the frequency changes of the input first spectrum, the second spectrum, and the third spectrum on the network can be reduced.

[0049] Among them, LSTM can perceive the timing information of speech and make full use of the current and historical information of speech.

[0050] Among them, the fully connected network can map the output of the LSTM neural network into a vector that matches the time-frequency MASK dimension.

[0051] In some embodiments, the first spectrum, the second spectrum and the third spectrum are input into the convolutional neural network in the speech separation model to obtain the first feature output by the convolutional neural network; the first feature is input into the long short-term memory network in the speech separation model to obtain the second feature output by the long short-term memory network; the second feature is input into the fully connected network in the speech separation model to obtain the time-frequency mask corresponding to the target speaker. Thus, the low-level features such as the first spectrum, the second spectrum and the third spectrum are first nonlinearly mapped by CNN to be converted into the high-level first feature, and then the first feature is input into LSTM to extract the time series feature to obtain the second feature output by LSTM, and finally the second feature is processed based on the fully connected network to obtain the time-frequency mask, so that the CNN and LSTM have clear division of labor, and the first spectrum, the second spectrum and the third spectrum are feature extracted in the two dimensions of time domain and frequency domain, thereby improving the accuracy of the acquired time-frequency mask.

[0052] Figure 2 A schematic diagram of a process for obtaining a target speech corresponding to a target speaker provided by an embodiment of the present disclosure. Figure 2 As shown, after obtaining the mixed signal, an adaptive beamforming technology is used to point the main lobe of the beam formed by the microphone array to the first direction and the side lobe of the beam to the second direction, so as to extract the first speech from the mixed speech, and to point the main lobe of the beam formed by the microphone array to the second direction and the side lobe of the beam to the first direction, so as to extract the second speech from the mixed speech; then, Fourier transform is performed on the first speech, the second speech and the third speech to obtain a first spectrum corresponding to the first speech, a second spectrum corresponding to the second speech and a third spectrum corresponding to the mixed speech, and the three spectra are spliced ​​in a preset order and input into CNN, the first feature output by CNN is input into LSTM to obtain the second feature output by LSTM, the second feature is input into fully connected network to obtain the time-frequency mask corresponding to the target speech, and finally the time-frequency mask is multiplied by the third spectrum to obtain a fourth spectrum corresponding to the target speech, and the fourth spectrum is inversely Fourier transformed to determine the target speech corresponding to the target speaker.

[0053] It should be noted that the beam signal directed to the target speaker, while suppressing the voice of the interfering speaker, will also affect the quality of the first voice directed to the target speaker (i.e., sound distortion), which may lead to the erroneous elimination of certain frequency points. Since the mixed voice is not distorted, the third spectrum corresponding to the mixed voice is input into the voice separation model together, which can make the voice separation model output a more accurate time-frequency mask, thereby making the distortion degree of the target voice finally separated smaller and the ability to suppress interference stronger. The second voice directed to the interfering speaker has a more obvious voice of the interfering speaker. If the energy of the second spectrum is strong and the energy of the first spectrum is weak at the same frequency point, the energy of the first spectrum is strong and the energy of the second spectrum is weak at another frequency point, so that the voice separation model can learn this energy difference, thereby better identifying the frequency point of the target signal. Therefore, based on the first spectrum, the second spectrum and the third spectrum, the target voice is separated from the mixed signal, and the quality of the acquired target voice can be provided.

[0054] In some embodiments, the energy difference between the first spectrum and the second spectrum can also be determined, and the frequency point corresponding to the energy difference greater than the threshold is determined as the frequency point corresponding to the target speech, and then based on the frequency point corresponding to the target speech, the time-frequency mask corresponding to the target speaker is determined, and the time-frequency mask is multiplied by the third spectrum to obtain a fourth spectrum corresponding to the target speech. Finally, based on the fourth spectrum, the target speech corresponding to the target speaker is determined.

[0055] It should be noted that when the energy corresponding to a frequency point of the first spectrum is greater than the energy corresponding to a frequency point of the second spectrum, the frequency point is the frequency point corresponding to the target speaker. Thus, each frequency point corresponding to the target speaker is determined in turn to obtain a time-frequency mask.

[0056] In some embodiments, an inverse Fourier transform is performed on the first spectrum to obtain a target speech corresponding to a target speaker.

[0057] In the disclosed embodiment, a first speech corresponding to the first direction where the target speaker is located and a second speech corresponding to the second direction where the interfering speaker is located are extracted from the mixed speech, and then a first spectrum corresponding to the first speech, a second spectrum corresponding to the second speech, and a third spectrum corresponding to the mixed speech are determined, and finally, the target speech corresponding to the target speaker is separated from the mixed speech according to the first spectrum, the second spectrum, and the third spectrum. Thus, the target speech corresponding to the target speaker can be accurately separated from the mixed signal by combining the first spectrum corresponding to the first speech in the first direction where the target speaker is located, the second spectrum corresponding to the second speech in the second direction where the interfering speaker is located, and the third spectrum corresponding to the mixed speech, thereby improving the quality of the separated target speech.

[0058] Figure 3is a flow chart of a speech processing method according to some embodiments of the present disclosure. Figure 3 As shown, the speech processing method is used in a terminal (such as a vehicle), comprising the following steps:

[0059] Step 301, controlling the vehicle to enter a sound zone communication mode.

[0060] In some embodiments, when the vehicle is driving on the road, the cabin noise is relatively loud due to wind noise, tire noise or other vehicles passing by. If the driver wants to talk to the passengers in the back row, it is difficult for the passengers in the back row to hear clearly what the driver is saying due to the loud noise, which leads to low communication efficiency, distracting the driver and increasing safety hazards. Therefore, a sound zone communication system can be deployed in the vehicle, that is, a microphone is arranged in the cabin, and the position of the microphone is not restricted. Generally, there will be a microphone near the seat, and a speaker is arranged near the seat. The position and type of the speaker are also not restricted. The area in the cabin is divided. There are currently several ways of division: the whole car sound zone; the front, middle, and rear sound zones; each seat is an independent sound zone. For vehicles with physical isolation between the cockpit and the passenger compartment, microphones and speakers need to be installed in the cockpit and the rear passenger compartment respectively. Therefore, the voice of any speaker can be extracted from the voice collected by the microphone and played in any sound zone in the cabin, which improves the call clarity and communication efficiency.

[0061] Figure 4 A schematic diagram of a multi-zone communication system provided by an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, microphones are deployed in front of each row of seats, and speakers are deployed on each seat. Front passengers can communicate with rear passengers, that is, the voice of the front passenger can be played from the speaker of the rear passenger's seat, and the voice of the rear passenger can be played from the speaker of the front passenger's seat.

[0062] Figure 5 A schematic diagram of another multi-zone communication provided by an embodiment of the present disclosure, such as Figure 5 As shown, microphones and speakers are deployed near each seat. The front passenger can communicate with the rear passenger. That is, the voice of the front passenger can be played from the speaker of the rear passenger's seat, and the voice of the rear passenger can be played from the speaker of the front passenger's seat.

[0063] Figure 6 A schematic diagram of another multi-zone communication provided by an embodiment of the present disclosure, such as Figure 6 As shown, microphones and speakers are deployed near each seat. The driver can communicate with the rear right passenger. That is, the driver's voice can be played from the speaker of the rear right passenger's seat, and the rear right passenger's voice can be played from the speaker of the driver's seat.

[0064] After the multi-zone communication system is deployed in the vehicle, it is possible to control whether the vehicle enters the multi-zone communication mode to achieve communication between the zones. In some embodiments, the multi-zone communication mode can be turned on by an intelligent voice assistant or by physical buttons on the vehicle to obtain a touch screen touch.

[0065] Step 302 : in response to receiving a sound zone communication request, obtaining the to-be-processed speech collected by the microphone array of the vehicle, wherein the sound zone communication request includes a first sound zone and a second sound zone that communicate with each other.

[0066] In some embodiments, after the vehicle enters the multi-zone communication mode, the passenger in the vehicle can trigger a zone communication request through the voice assistant. For example, the zone communication request can be "establish communication between the driver and the right side of the rear seat", where the driver's zone can be the first zone and the right side of the rear seat can be the second zone.

[0067] In some embodiments, passengers in the vehicle can also trigger a sound zone communication request by touching the touch screen.

[0068] Figure 7 A schematic diagram of a display interface provided by an embodiment of the present disclosure, such as Figure 7 As shown, the vehicle and each seat in the vehicle can be displayed on the display screen, so that the sound zone communication request can be triggered by touching the seat on the touch screen. For example, the driver communicates with the rear right passenger, and the driver can touch the seat corresponding to the driver and the seat corresponding to the rear right side to trigger the sound zone communication request. The sound zone where the driver is located can be the first sound zone, and the sound zone where the rear right side is located can be the second sound zone.

[0069] In some embodiments, the voice zone where the speaker who triggers the voice zone communication request is located can be determined as the first voice zone.

[0070] In some embodiments, two-way or one-way communication between audio zones can be achieved. For example, if the passenger in the main driving car clicks the rear right seat, it indicates one-way communication, that is, the sound of the main driving car can be played from the rear right speaker, while the sound of the rear right cannot be played from the main driving car speaker. If the passenger in the main driving car double-clicks the rear right seat, it indicates two-way communication, that is, the sound of the main driving car can be played from the rear right speaker, and the sound of the rear right can also be played from the main driving car speaker.

[0071] Step 303 , localize the sound source of the speech to be processed to obtain the number of sound sources contained in the speech to be processed and the sound zone where each sound source is located.

[0072] Step 304: if there are multiple sound sources in the speech to be processed and there are sound sources in the first sound range and / or the second sound range, determine that the speech to be processed is a mixed speech, wherein the speaker of the first sound range and / or the second sound range is the target speaker.

[0073] In some embodiments, if the first sound zone communicates with the second sound zone unidirectionally, and the sound sources of the first sound zone and the second sound zone exist in the mixed speech, the speaker of the first sound zone can be used as the target speaker, and the speaker of the second sound zone can be used as the target speaker.

[0074] In some embodiments, if the second audio zone communicates with the first audio zone in a two-way manner, and if there are sound sources of the first audio zone and the second audio zone, there are still sound sources of other audio zones, the speakers of the first audio zone and the second audio zone can be determined as target speakers, so that the target speech of the target speakers of the first audio zone and the second audio zone can be obtained at the same time.

[0075] In some embodiments, if the second sound zone communicates with the first sound zone in a two-way manner, when there are sound sources of the first sound zone and the second sound zone in the mixed speech, the speaker of the first sound zone can be used as the target speaker, and the speaker of the second sound zone can be used as the interference speaker to separate the target speech corresponding to the speaker of the first sound zone from the mixed speech, and then the speaker of the second sound zone can be used as the target speaker, and the speaker of the first sound zone can be used as the interference speaker to separate the target speech corresponding to the speaker of the second sound zone from the mixed speech. Thus, when the speaker of the first sound zone is the target speaker, the speaker of the second sound zone can be used as the interference speaker to obtain the speech corresponding to the speaker of the first sound zone from the mixed speech, and when the speaker of the second sound zone is the target speaker, the speaker of the first sound zone can be used as the interference speaker to obtain the speech corresponding to the speaker of the second sound zone from the mixed speech, so that the speech of the speaker corresponding to each sound zone can be obtained more accurately.

[0076] Step 305 , extracting from the mixed speech a first speech corresponding to a first direction where the target speaker is located and a second speech corresponding to a second direction where the interfering speaker is located.

[0077] In this embodiment, the first direction where the target speaker is located may be the sound zone where the target speaker is located. For example, if the target speaker is a speaker in the first sound zone, the first direction where the target speaker is located may be the first sound zone. Thus, the first direction can be quickly determined.

[0078] Step 306: determine a first spectrum corresponding to the first speech, a second spectrum corresponding to the second speech, and a third spectrum corresponding to the mixed speech.

[0079] Step 307 : Separate the target speech corresponding to the target speaker from the mixed speech according to the first spectrum, the second spectrum and the third spectrum.

[0080] The specific implementation forms of steps 305 to 307 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0081] Step 308: Play the target speech corresponding to the target speaker in the first audio zone through the speaker in the second audio zone; and / or play the target speech corresponding to the target speaker in the second audio zone through the speaker in the first audio zone.

[0082] In some embodiments, when the first audio zone communicates unidirectionally with the second audio zone, the target speech corresponding to the target speaker of the first audio zone is played through the speaker of the second audio zone.

[0083] In some embodiments, if the second audio zone communicates unidirectionally with the first audio zone, the target speech corresponding to the target speaker of the first audio zone is played through the speaker of the second audio zone.

[0084] In some embodiments, if the second audio zone communicates bidirectionally with the first audio zone, the target speech corresponding to the target speaker of the first audio zone is played through the speaker of the second audio zone; and the target speech corresponding to the target speaker of the second audio zone is played through the speaker of the first audio zone.

[0085] In the embodiment of the present disclosure, when a vehicle is deployed with a multi-zone communication system, the vehicle can be controlled to enter a zone communication mode, and when a zone communication request is received, the speech to be processed collected by the microphone array of the vehicle is obtained, wherein the zone communication request includes the first zone and the second zone of communication; sound source positioning is performed on the speech to be processed to obtain the number of sound sources contained in the speech to be processed and the zone where each sound source is located, and when there are multiple sound sources in the speech to be processed and there are sound sources in the first zone and / or the second zone, it is determined that the speech to be processed is a mixed speech, wherein the speaker in the first zone and / or the second zone is the target speaker, and then the target speech corresponding to the target speaker is separated from the mixed speech, and finally the target speech corresponding to the target speaker in the first zone is played through the speaker in the second zone, and / or the target speech corresponding to the target speaker in the second zone is played through the speaker in the first zone. Therefore, when a multi-zone communication system is deployed in a vehicle, the target voice corresponding to the target speaker in the communication sound zone can be separated from the mixed voice based on the first voice of the target speaker in the communication sound zone and the second voice of the interfering speaker, and played on the speaker in the corresponding listener sound zone, thereby improving the quality of the target voice separated from the mixed voice and further improving the quality of the sound zone communication.

[0086] In some embodiments, the cockpit multi-zone communication system uses a microphone near the cockpit speaker to pick up the speaker's voice, and after signal processing, it is played out by a speaker near the listener. This path is the forward or direct path (feedforward / direct path). In addition, since the microphone and the speaker are distributed in the cockpit and belong to the same acoustic space, there is an acoustic path from the speaker to the microphone, which is called the feedback path (feedback path), that is, the microphone, the speaker, and the limited space of the cockpit form a closed-loop electroacoustic system. This requires suppressing and eliminating the signal on the feedback path (feedback cancellation), and reasonably controlling the forward gain to keep the cockpit multi-zone communication system in a stable state. Therefore, a time domain adaptive echo cancellation algorithm and a frequency domain adaptive echo cancellation algorithm can be used to perform echo cancellation on the collected mixed signal. The time domain adaptive echo cancellation algorithm includes: Least Mean Square (LMS), Normalized Least Mean Square (NLMS), Adaptive Fast Transversal Filter (AFTF), etc. The frequency domain adaptive echo cancellation algorithm includes: Frequency domain Adaptive Filter (FDAF), Partioned Block Frequency Domain Adaptive Filter (PBFDAF), Adaptive Kalman Filter (AKF), etc.

[0087] In some embodiments, automatic gain control (AGC) technology can also be used to keep the gain of the target speech within a preset reasonable range to avoid fluctuations.

[0088] In some embodiments, the target speech may also be subjected to equalizer (EQ) or dynamic range control (DRC), thereby providing compression and amplification capabilities, which may make the voice sound softer or louder.

[0089] In some embodiments, by performing channel weighting on the loudspeaker in the target sound zone that plays the target voice, the sound played by the loudspeaker is made directional, and the sound of the loudspeaker is directed toward the passengers in the target sound zone, thereby preventing the sound in the target sound zone from disturbing the passengers in non-target sound zones.

[0090] In some embodiments, since the ambient noise in the cabin is sometimes loud and sometimes soft, the volume of the target voice played can be adaptively adjusted according to the size of the ambient noise recorded by the microphone to avoid the driver or other passengers manually adjusting the volume.

[0091] In order to implement the above embodiment, the present disclosure also proposes a speech processing device.

[0092] Figure 8 is a block diagram of a speech processing device according to some embodiments of the present disclosure. Figure 8 , the device comprises:

[0093] An acquisition module 801 is used to acquire mixed speech, wherein the mixed speech includes a target speech corresponding to a target speaker and an interference speech corresponding to an interference speaker;

[0094] An extraction module 802 is used to extract a first speech corresponding to a first direction where a target speaker is located and a second speech corresponding to a second direction where an interfering speaker is located from the mixed speech;

[0095] A determination module 803 is used to determine a first spectrum corresponding to the first speech, a second spectrum corresponding to the second speech, and a third spectrum corresponding to the mixed speech;

[0096] The separation module 804 is used to separate the target speech corresponding to the target speaker from the mixed speech according to the first spectrum, the second spectrum and the third spectrum.

[0097] In some embodiments, the separation module 804 is used to:

[0098] Inputting the first spectrum, the second spectrum and the third spectrum into a speech separation model to obtain a time-frequency mask corresponding to the target speaker;

[0099] Multiplying the time-frequency mask by the third spectrum to obtain a fourth spectrum corresponding to the target speech;

[0100] Based on the fourth frequency spectrum, a target speech corresponding to the target speaker is determined.

[0101] In some embodiments, the separation module 804 is used to:

[0102] Inputting the first spectrum, the second spectrum and the third spectrum into a convolutional neural network in a speech separation model to obtain a first feature output by the convolutional neural network;

[0103] Inputting the first feature into a long short-term memory network in a speech separation model to obtain a second feature output by the long short-term memory network;

[0104] The second feature is input into the fully connected network in the speech separation model to obtain the time-frequency mask corresponding to the target speaker.

[0105] In some embodiments, the separation module 804 is used to:

[0106] determining an energy difference between the first spectrum and the second spectrum;

[0107] The frequency point corresponding to the energy difference greater than the threshold is determined as the frequency point corresponding to the target speech;

[0108] Based on the frequency points corresponding to the target speech, determine the time-frequency mask corresponding to the target speaker;

[0109] Multiplying the time-frequency mask by the third spectrum to obtain a fourth spectrum corresponding to the target speech;

[0110] Based on the fourth frequency spectrum, a target speech corresponding to the target speaker is determined.

[0111] In some embodiments, the extraction module 802 is used to:

[0112] Performing sound source localization on the mixed speech to obtain a first direction and a second direction;

[0113] Directing a main lobe of a beam formed by a microphone array in a first direction and a side lobe of the beam in a second direction, so as to extract a first speech from a mixed speech, wherein the microphone array is used to collect the mixed speech;

[0114] The main lobe of the beam formed by the microphone array is directed to the second direction, and the side lobe of the beam is directed to the first direction, so as to extract the second speech from the mixed speech.

[0115] In some embodiments, the acquisition module 802 is used to:

[0116] Control the vehicle to enter the audio zone communication mode;

[0117] In response to receiving a sound zone communication request, obtaining the to-be-processed speech collected by the microphone array of the vehicle, wherein the sound zone communication request includes a first sound zone and a second sound zone of the communication;

[0118] Perform sound source localization on the speech to be processed to obtain the number of sound sources contained in the speech to be processed and the sound zone where each sound source is located;

[0119] When there are multiple sound sources in the speech to be processed and there are sound sources in the first sound range and / or the second sound range, the speech to be processed is determined to be a mixed speech, wherein the speaker of the first sound range and / or the second sound range is the target speaker.

[0120] In some embodiments, the system further includes a processing module for:

[0121] When there are sound sources in the first and second ranges in the mixed speech, the speaker in the first range is taken as the target speaker, the speaker in the second range is taken as the interference speaker, and the target speech corresponding to the speaker in the first range is separated from the mixed speech;

[0122] The speaker in the second range is taken as the target speaker, the speaker in the first range is taken as the interference speaker, and the target speech corresponding to the speaker in the second range is separated from the mixed speech.

[0123] In some embodiments, a control module is further included for:

[0124] Playing the target speech corresponding to the target speaker in the first audio zone through the speaker in the second audio zone; and / or,

[0125] The target speech corresponding to the target speaker in the second sound zone is played through the loudspeaker in the first sound zone.

[0126] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0127] The speech processing device of the disclosed embodiment extracts the first speech corresponding to the first direction where the target speaker is located and the second speech corresponding to the second direction where the interfering speaker is located from the mixed speech, then determines the first spectrum corresponding to the first speech, the second spectrum corresponding to the second speech, and the third spectrum corresponding to the mixed speech, and finally separates the target speech corresponding to the target speaker from the mixed speech according to the first spectrum, the second spectrum and the third spectrum. Thus, the target speech corresponding to the target speaker can be accurately separated from the mixed signal by combining the first spectrum corresponding to the first speech in the first direction where the target speaker is located, the second spectrum corresponding to the second speech in the second direction where the interfering speaker is located, and the third spectrum corresponding to the mixed speech, thereby improving the quality of the separated target speech.

[0128] The present disclosure also provides an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to: implement the steps of the speech processing method provided by the present disclosure.

[0129] In some embodiments, the electronic device may be a vehicle.

[0130] Fig. 9 is a block diagram of a vehicle 900 according to an exemplary embodiment. For example, the vehicle 900 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicles. The vehicle 900 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0131] Reference Fig. 9 , the vehicle 900 may include various subsystems, for example, an infotainment system 910, a perception system 920, a decision control system 930, a drive system 940, and a computing platform 950. The vehicle 900 may also include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and each component of the vehicle 900 may be interconnected by wire or wireless means.

[0132] In some embodiments, the infotainment system 910 may include a communication system, an entertainment system, and a navigation system, etc. The perception system 920 may include several sensors for sensing information about the environment around the vehicle 900. For example, the perception system 920 may include a global positioning system (the global positioning system may be a GPS system, a Beidou system, or other positioning systems), an inertial measurement unit (IMU), a laser radar, a millimeter wave radar, an ultrasonic radar, and a camera.

[0133] The decision control system 930 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system. The drive system 940 may include components that provide power movement for the vehicle 900. In one embodiment, the drive system 940 may include an engine, an energy source, a transmission system, and wheels. The engine may be a combination of one or more of an internal combustion engine, an electric motor, and an air compression engine. The engine can convert the energy provided by the energy source into mechanical energy.

[0134] Some or all functions of the vehicle 900 are controlled by a computing platform 950. The computing platform 950 may include at least one processor 951 and a memory 952, and the processor 951 may execute instructions 953 stored in the memory 952.

[0135] The processor 951 may be any conventional processor, such as a commercially available CPU. The processor may also include a graphics processor (Graphic Process Unit, GPU), a field programmable gate array (Field Programmable Gate Array, FPGA), a system on chip (System on Chip, SOC), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC) or a combination thereof.

[0136] The memory 952 may be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0137] In addition to the instructions 953, the memory 952 may also store data, such as road maps, route information, vehicle location, direction, speed, etc. The data stored in the memory 952 may be used by the computing platform 950. In the disclosed embodiment, the processor 951 may execute the instructions 953 to complete all or part of the steps of the above-mentioned speech processing method.

[0138] The present disclosure also provides a computer-readable storage medium on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the speech processing method provided by the present disclosure are implemented.

[0139] In addition, the word "exemplary" is used herein to indicate serving as an example, instance, or diagram. Any aspect or design described as "exemplary" in this article is not necessarily understood to be advantageous compared to other aspects or designs. On the contrary, the use of the word exemplary is intended to present concepts in a specific way. As used herein, the term "or" is intended to represent an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X applies A or B" is intended to represent any one of the natural inclusive arrangements. That is, if X applies A; X applies B; or X applies both A and B, "X applies A or B" is satisfied under any of the aforementioned examples. In addition, unless otherwise specified or clearly pointed to a singular form from the context, the articles "one" and "an" as used in this application and the appended claims are generally understood to mean "one or more".

[0140] Likewise, although the present disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art after reading and understanding the specification and drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components (e.g., elements, resources, etc.) described above, unless otherwise noted, the terms used to describe such components are intended to correspond to any component (functionally equivalent) that performs the specific functions of the described components, even if the structure is not equivalent to the disclosed structure. In addition, although the specific features of the present disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations as may be desired and conducive to any given or specific application. In addition, with respect to "including", "having", "having", "having", or variations thereof used in the specific embodiments or claims, such terms are intended to be inclusive in a manner similar to the term "comprising".

[0141] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0142] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A speech processing method, characterized in that: include: Acquire a mixed speech, wherein the mixed speech includes a target speech corresponding to a target speaker and an interfering speech corresponding to an interfering speaker; Extracting from the mixed speech a first speech corresponding to a first direction where the target speaker is located and a second speech corresponding to a second direction where the interfering speaker is located; Determine a first spectrum corresponding to the first speech, a second spectrum corresponding to the second speech, and a third spectrum corresponding to the mixed speech; Inputting the first spectrum, the second spectrum and the third spectrum into a speech separation model to obtain a time-frequency mask corresponding to the target speaker; Multiplying the time-frequency mask by the third spectrum to obtain a fourth spectrum corresponding to the target speech; The target speech corresponding to the target speaker is determined based on the fourth frequency spectrum.

2. The method according to claim 1, characterized in that The step of inputting the first spectrum, the second spectrum, and the third spectrum into a speech separation model to obtain a time-frequency mask corresponding to the target speaker includes: Inputting the first spectrum, the second spectrum and the third spectrum into the convolutional neural network in the speech separation model to obtain a first feature output by the convolutional neural network; Inputting the first feature into a long short-term memory network in the speech separation model to obtain a second feature output by the long short-term memory network; The second feature is input into a fully connected network in the speech separation model to obtain a time-frequency mask corresponding to the target speaker.

3. The method according to claim 1, characterized in that: The extracting, from the mixed speech, a first speech corresponding to a first direction where the target speaker is located and a second speech corresponding to a second direction where the interfering speaker is located, comprises: Performing sound source localization on the mixed speech to obtain the first direction and the second direction; Directing a main lobe of a beam formed by a microphone array toward the first direction and a side lobe of the beam toward the second direction, so as to extract the first speech from the mixed speech, wherein the microphone array is used to collect the mixed speech; The main lobe of the beam formed by the microphone array is directed toward the second direction, and the side lobe of the beam is directed toward the first direction, so as to extract the second speech from the mixed speech.

4. The method according to claim 1, characterized in that: The obtaining of mixed speech comprises: Control the vehicle to enter the audio zone communication mode; In response to receiving a sound zone communication request, obtaining the to-be-processed speech collected by the microphone array of the vehicle, wherein the sound zone communication request includes a first sound zone and a second sound zone of communication; Performing sound source localization on the speech to be processed to obtain the number of sound sources contained in the speech to be processed and the sound zone where each sound source is located; When there are multiple sound sources in the speech to be processed and there are sound sources in the first sound zone and / or the second sound zone, the speech to be processed is determined to be the mixed speech, wherein the speaker of the first sound zone and / or the second sound zone is the target speaker.

5. The method according to claim 4, characterized in that Also includes: When the mixed speech contains sound sources of the first sound range and the second sound range, a speaker of the first sound range is used as a target speaker, a speaker of the second sound range is used as an interference speaker, and a target speech corresponding to the speaker of the first sound range is separated from the mixed speech; The speaker in the second range is used as a target speaker, the speaker in the first range is used as an interfering speaker, and the target speech corresponding to the speaker in the second range is separated from the mixed speech.

6. The method according to claim 4, characterized in that After determining the target speech corresponding to the target speaker based on the fourth spectrum, the method further includes: Playing the target speech corresponding to the target speaker in the first audio zone through the speaker in the second audio zone; and / or, The target speech corresponding to the target speaker in the second sound zone is played through the loudspeaker in the first sound zone.

7. A speech processing device, characterized in that: The device comprises: An acquisition module, configured to acquire a mixed speech, wherein the mixed speech includes a target speech corresponding to a target speaker and an interfering speech corresponding to an interfering speaker; An extraction module, configured to extract, from the mixed speech, a first speech corresponding to a first direction where the target speaker is located and a second speech corresponding to a second direction where the interfering speaker is located; A determination module, configured to determine a first spectrum corresponding to the first speech, a second spectrum corresponding to the second speech, and a third spectrum corresponding to the mixed speech; The separation module is used to input the first spectrum, the second spectrum and the third spectrum into a speech separation model to obtain a time-frequency mask corresponding to the target speaker; multiply the time-frequency mask by the third spectrum to obtain a fourth spectrum corresponding to the target speech; and determine the target speech corresponding to the target speaker based on the fourth spectrum.

8. An electronic device, characterized in that: include: processor; A memory for storing processor-executable instructions; wherein the processor is configured to: implement the steps of the method described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a mobile terminal, the steps of the method described in any one of claims 1 to 6 are implemented.

Images