Voice separation method, device and system, electronic equipment and storage medium

Abstract

The invention provides a voice separation method, device and system, electronic equipment and a storage medium, and the method comprises the steps: carrying out the voice separation between voice regions and the voice separation in the voice regions on the voice signals of a plurality of voice regions based on the frequency spectrum information and phase information of the voice signals of the plurality of voice regions on a target vehicle; and determining voice separation results of a plurality of voice regions based on a voice interval separation result obtained by voice interval voice separation and a voice region internal separation result obtained by voice region internal separation, thereby overcoming the defect of poor vehicle-mounted voice separation effect in a severe scene in a traditional scheme. Spatial information can be better acquired through signal acquisition of a plurality of microphone arrays, two times of voice separation with different precisions are performed on the basis, and results are fused, so that more accurate voice separation is realized, accurate and reliable voice separation results can be obtained, the wake-up rate of the vehicle-mounted interaction system is effectively improved, and the user experience is improved. Therefore, the voice interaction experience of the user is optimized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a speech separation method, device, system, electronic device and storage medium. Background Art

[0002] With the development of intelligent vehicles, in-vehicle interactive systems have gradually become an important part of modern vehicles, providing drivers and passengers with a more convenient and intelligent interactive experience. The system controls various functions in the car by recognizing the wake-up words and commands of passengers. However, in actual applications, the in-car environment is complex and changeable, including interference factors such as car driving noise, wind noise, and multiple people talking at the same time, which poses great challenges to the voice separation of the speakers in the car.

[0003] At present, speech separation in vehicles mostly relies on a single microphone array, that is, speech separation is performed through a single microphone array installed in the car. This method often performs unsatisfactorily in harsh scenarios with low signal-to-noise ratio or other severe interference. In other words, in a low signal-to-noise ratio environment, the signal quality of a single microphone array is greatly reduced, and accurate speech separation cannot be achieved based on this. Even if separation can be completed, the separated signal is easily distorted, which affects the subsequent wake-up and recognition process, and then leads to a significant decrease in the user's voice interaction experience. Summary of the invention

[0004] The present invention provides a speech separation method, device, system, electronic device and storage medium, which are used to solve the problems in the prior art that the vehicle-mounted speech separation effect in harsh scenes is poor, the speech separation and positioning cannot be accurately performed, and the subsequent interaction process is affected. The speech separation effect in harsh scenes is improved, accurate speech separation is achieved, and interaction efficiency is improved.

[0005] The present invention provides a speech separation method, comprising: Acquire voice signals of multiple sound zones on the target vehicle; Based on the frequency spectrum information and phase information of the speech signals of the multiple sound zones, performing inter-sound zone speech separation and intra-sound zone speech separation on the speech signals of the multiple sound zones; Based on the between-tone separation result obtained by the between-tone separation and the within-tone separation result obtained by the within-tone separation, the speech separation results of the plurality of tone zones are determined.

[0006] According to a speech separation method provided by the present invention, based on the spectrum information and phase information of the speech signals of the multiple sound zones, performing inter-sound zone speech separation and intra-sound zone speech separation on the speech signals of the multiple sound zones, the method comprises: Determine speech features of the plurality of sound zones based on the frequency spectrum information and phase information of the speech signals of the plurality of sound zones, and perform inter-speech separation on the speech signals of the plurality of sound zones based on the speech features of the plurality of sound zones; Each of the multiple sound zones is divided into sub-zones to obtain multiple sub-zones under each sound zone, and based on the speech features of the multiple sub-zones, the speech signal of the corresponding sound zone is subjected to intra-zone speech separation.

[0007] According to a speech separation method provided by the present invention, the inter-sound zone separation result includes the separation results of each of the multiple sound zones, and the intra-sound zone separation result includes the separation results of each of the multiple sub-sound zones under the corresponding sound zone; Determining the speech separation results of the plurality of speech zones based on the between-sound zone speech separation result obtained by the within-sound zone speech separation and the within-sound zone speech separation result obtained by the within-sound zone speech separation comprises: From the intra-range separation results of each range, the separation results of the target sub-range are obtained by screening; Based on the separation result of the target consonant area and the separation result of the area corresponding to the target consonant area, a speech separation result of the area corresponding to the target consonant area is determined.

[0008] According to a speech separation method provided by the present invention, the method of determining the speech separation result of the corresponding sound area based on the separation result of the target consonant area and the separation result of the sound area corresponding to the target consonant area includes: Determine a target separation result of the corresponding sound area based on the separation result of the target subtone area and the separation result of the sound area corresponding to the target subtone area; Narrowband enhancement is performed based on the target separation result of the corresponding sound area to obtain the speech separation result of the corresponding sound area.

[0009] According to a speech separation method provided by the present invention, the narrowband enhancement is performed based on the target separation result of the corresponding sound area to obtain the speech separation result of the corresponding sound area, including: Determine a covariance matrix of the corresponding tone area based on the target separation result of the corresponding tone area and the speech signal; Determine a narrowband signal corresponding to the tone range based on a covariance matrix corresponding to the tone range; Based on the narrowband signal of the corresponding sound area and the target separation result, narrowband enhancement is performed to obtain the speech separation result of the corresponding sound area.

[0010] According to a speech separation method provided by the present invention, the determining of speech features of the multiple sound zones based on the spectrum information and phase information of the speech signals of the multiple sound zones includes: Performing fixed beam forming on the speech signals of the multiple sound zones to obtain fixed beams of the multiple sound zones; Based on the frequency spectrum information and phase information corresponding to the fixed beams of the multiple sound zones, speech features of the multiple sound zones are determined.

[0011] The present invention also provides a speech separation device, comprising: An acquisition unit, used for acquiring voice signals of multiple sound zones on a target vehicle; A separation unit, configured to perform inter-sound zone speech separation and intra-sound zone speech separation on the speech signals of the plurality of sound zones based on the spectrum information and phase information of the speech signals of the plurality of sound zones; The fusion unit is used to determine the speech separation results of the multiple sound zones based on the between-sound zone separation results obtained by the between-sound zone speech separation and the within-sound zone separation results obtained by the within-sound zone speech separation.

[0012] The present invention also provides a vehicle-mounted interactive system, comprising a plurality of microphone arrays, a processor and a display; the plurality of microphone arrays are respectively arranged in a plurality of sound zones on a target vehicle; The processor is used to obtain voice signals of multiple sound zones collected by the multiple microphone arrays on the target vehicle, and perform inter-sound zone voice separation and intra-sound zone voice separation on the voice signals of the multiple sound zones based on the spectrum information and phase information of the voice signals of the multiple sound zones; determine the voice separation results of the multiple sound zones based on the inter-sound zone separation results obtained by the inter-sound zone voice separation and the intra-sound zone separation results obtained by the intra-sound zone voice separation, perform voice wake-up and voice interaction based on the voice separation results of the multiple sound zones, obtain reply content, and control the display to display the reply content.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the speech separation method as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the speech separation method as described above is implemented.

[0015] The speech separation method, device, system, electronic device and storage medium provided by the present invention extract the spectrum information and phase information of the speech signals of multiple sound zones on the target vehicle, and based on this, perform inter-sound zone speech separation and intra-sound zone speech separation on the speech signals of the multiple sound zones, and determine the speech separation results of the multiple sound zones based on the inter-sound zone separation results obtained by the inter-sound zone speech separation and the intra-sound zone separation results obtained by the intra-sound zone speech separation, thereby overcoming the defects of the traditional scheme that the vehicle-mounted speech separation effect is not good under harsh scenarios, the speech separation and positioning cannot be performed accurately, and the subsequent interaction process is affected. Signal collection through multiple microphone arrays can better obtain spatial information, and on this basis, two speech separations with different precisions are performed and the results are fused, thereby achieving more precise speech separation, and being able to obtain accurate and reliable speech separation results, effectively improving the wake-up rate of the vehicle-mounted interactive system and the positioning accuracy, thereby optimizing the user's voice interaction experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a flow chart of the speech separation method provided by the present invention; Figure 2 It is an overall flow chart of the speech separation method provided by the present invention; Figure 3 It is a structural schematic diagram of the speech separation device provided by the present invention; Figure 4 is a structural schematic diagram of the vehicle-mounted interactive system provided by the present invention; Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] At present, in-vehicle voice separation is mostly performed through a single microphone array distributed in the car. Although this solution can achieve voice separation, its separation effect is often poor when facing a bad scene with low signal-to-noise ratio or a lot of interference; in addition, the separated signal is easily distorted, resulting in the inability to wake up, which greatly reduces the user's voice interaction experience.

[0020] In this regard, the present invention provides a speech separation method, which aims to collect signals through multiple microphone arrays to better obtain spatial information, and use the speech signals of multiple sound zones collected by the multiple microphone arrays to perform speech separation twice with different accuracies, and fuse the two separation results, so as to achieve more precise speech separation and obtain accurate speech separation results, thereby solving the problem of poor vehicle-mounted speech separation effect under current harsh scenarios and effectively improving the wake-up rate and positioning rate of the vehicle-mounted interactive system.

[0021] Figure 1 It is a flow chart of the speech separation method provided by the present invention, such as Figure 1 As shown, the method can be applied to the in-vehicle interactive system, and the specific execution subject can be a processor in the in-vehicle interactive system. The method includes: Step 110, acquiring voice signals of multiple sound zones on the target vehicle; Step 120, performing inter-sound zone speech separation and intra-sound zone speech separation on the speech signals of the multiple sound zones based on the spectrum information and phase information of the speech signals of the multiple sound zones; Step 130 , based on the inter-sound zone separation result obtained by the inter-sound zone speech separation and the intra-sound zone separation result obtained by the intra-sound zone speech separation, determine the speech separation results for the plurality of sound zones.

[0022] Specifically, considering the current use of a single array for speech separation, when encountering harsh scenarios with low signal-to-noise ratio or many interference factors, speech separation cannot be performed accurately, and the separation effect is poor, which affects the subsequent voice interaction process and thus leads to a decline in user experience.

[0023] Based on this, in an embodiment of the present invention, it is proposed that multiple microphone arrays can be deployed on the target vehicle, and signals can be collected through microphone arrays in multiple sound zones. The collected voice signals in the multiple sound zones can be used to perform speech separation twice, and the separation results can be fused, thereby effectively improving the speech separation effect and obtaining accurate and reliable speech separation results, thereby improving the wake-up rate of the voice interaction process and further enhancing the user's voice interaction experience.

[0024] It is understandable that in the actual application process, before performing speech separation, it is first necessary to determine the speech signal to be separated. In consideration of the problems existing in the current speech separation based on a single array, in an embodiment of the present invention, multiple microphone arrays are arranged to perform speech separation through the speech signals corresponding to the multiple microphone arrays. Specifically, here, microphone arrays can be arranged in multiple sound zones on the target vehicle in advance, and the speech signals in the vehicle are collected through multiple microphone arrays, so that the speech signals in multiple sound zones on the target vehicle can be obtained. It should be noted that because the interior of the vehicle is usually not soundproof and the sound zones are not completely isolated, the speech signals collected by the microphone arrays in each sound zone do not completely correspond to the sound zone in which it is located, and often contain sound signals of other sound zones, such as the sound signals of speakers in other sound zones, noise signals, etc. It is precisely because of this that in an embodiment of the present invention, it is necessary to perform speech separation on the collected speech signals to obtain accurate separation signals for each sound zone (which do not contain signals of other sound zones).

[0025] Here, the sound zones on the target vehicle can be divided according to the seat positions. For example, for a conventional four-seater car, it can be divided into the main driving sound zone, the co-pilot sound zone, the main driving rear sound zone and the co-pilot rear sound zone. For a five-seater or seven-seater car, it can be divided into the main driving sound zone, the co-pilot sound zone, the middle sound zone, the left rear sound zone and the right rear sound zone. It can also be divided according to the structure of the vehicle, the interior environment, the layout, etc., which is not specifically limited in the embodiment of the present invention. Among them, one microphone array or multiple microphone arrays can be arranged in a sound zone, which is not specifically limited in the embodiment of the present invention. As a preference, in order to save costs, improve processing efficiency and ensure separation effect, a microphone array is arranged in each sound zone in the embodiment of the present invention, and signal collection is performed accordingly to obtain the voice signal of each sound zone.

[0026] Compared with the traditional array, the distributed microphone array (a collection of multiple microphone arrays) used in the embodiment of the present invention is not constrained by the array. Not only can each array process the received sound signal independently, but the overall performance will not be reduced due to the failure of one or several array elements. Using the distributed microphone array for speech separation, due to the participation of multiple arrays, fast and accurate speech separation and sound source localization can be achieved in harsh environments.

[0027] Furthermore, after obtaining the speech signals of multiple sound zones, in an embodiment of the present invention, speech separation can be performed based on this to obtain separation results. That is, the speech signals of multiple sound zones are subjected to speech separation using the spectrum information and phase information of the speech signals of multiple sound zones, thereby obtaining separation results of multiple sound zones. In order to achieve more precise speech separation and obtain more accurate and reliable speech separation results, the speech separation process here can be divided into two levels, namely, speech separation between sound zones and speech separation within sound zones. Among them, speech separation between sound zones is to perform speech separation between sound zones for different sound zones, and obtain separation results between sound zones; speech separation within sound zones is to perform speech separation within each sound zone, and obtain separation results within sound zones.

[0028] Specifically, here, the information of the speech signal of each sound zone can be extracted first to extract the information about the spectrum, phase, etc. in the speech signal, such as the logarithmic power spectrum, phase difference, etc., thereby obtaining the spectrum information and phase information of the speech signal of each sound zone. Then, speech separation can be performed based on the spectrum information and phase information, specifically, the spectrum information and phase information of the speech signals of multiple sound zones are used to perform the speech separation between sound zones and the speech separation within sound zones respectively, so as to separate the signals of different sound zones, thereby obtaining the separation signals of each sound zone, i.e., the separation result between sound zones; at the same time, the signals of different areas / positions in each sound zone are separated, thereby obtaining the separation signals of each area / position in each sound zone, i.e., the separation result within sound zones.

[0029] After that, the final separation result can be determined based on the interval separation result and the intra-zone separation result. That is, based on the interval separation result and the intra-zone separation result obtained by the interval speech separation and the intra-zone speech separation, signal fusion is performed to obtain the final separation results of multiple sound zones, and the separation signal is the final speech separation result of multiple sound zones. Specifically, the interval separation result and the intra-zone separation result obtained by the two speech separations are fused here to achieve more accurate speech separation through the fusion of the separation signals, and finally the speech separation results of multiple sound zones can be obtained.

[0030] Furthermore, after obtaining the speech separation results of multiple sound zones, in the embodiment of the present invention, in-vehicle voice interaction can be performed according to the speech separation results, and the reply output of the in-vehicle interactive system can be obtained. That is, voice wake-up can be performed according to the speech separation results of multiple sound zones, and the sound source can be located according to the energy, so as to carry out targeted voice interaction, respond to the input voice, and generate reply information, which is output and displayed so that the corresponding personnel on the target vehicle can know the reply information of the system in time, realize voice interaction on the target vehicle, improve the wake-up rate of the system backend, and optimize the user's voice interaction experience.

[0031] The following takes a target vehicle equipped with a distributed array consisting of four microphone arrays as an example to illustrate the speech separation process of the speech signal on it: The target vehicle is equipped with four microphone arrays, which are respectively arranged above the four seats (the main driver's seat, the co-driver's seat, the rear seat of the main driver and the rear seat of the co-driver). On the target vehicle, the distributed microphone array can pick up multi-channel voice signals, that is, voice signals of multiple sound zones, and then the voice signals of multiple sound zones can be used for voice separation. Specifically, the voice separation between sound zones is first performed according to the voice signals of multiple sound zones, and the voice separation within the sound zone is performed using the voice signals collected by the single array, and the separation results between sound zones and within sound zones are obtained respectively. After that, the separation results obtained from the two voice separations can be fused to obtain accurate and reliable separation signals for multiple sound zones. In this way, accurate voice separation can be achieved in harsh scenes, and accurate separation signals of each sound zone can be extracted from the complex vehicle environment for subsequent voice wake-up and sound source positioning, so as to better realize in-vehicle voice interaction.

[0032] The speech separation method provided by the present invention extracts the spectrum information and phase information of the speech signals of multiple sound zones on the target vehicle, and based on this, performs inter-sound zone speech separation and intra-sound zone speech separation on the speech signals of the multiple sound zones, and determines the speech separation results of the multiple sound zones based on the inter-sound zone separation results obtained by the inter-sound zone speech separation and the intra-sound zone separation results obtained by the intra-sound zone speech separation, thereby overcoming the defects of the traditional scheme that the vehicle-mounted speech separation effect is not good under harsh scenes, the speech separation and positioning cannot be performed accurately, and the subsequent interaction process is affected. Signal collection through multiple microphone arrays can better obtain spatial information, and on this basis, two speech separations with different precisions are performed, and the results are fused, thereby achieving more accurate speech separation, and being able to obtain accurate and reliable speech separation results, effectively improving the wake-up rate of the vehicle-mounted interactive system and the positioning accuracy, thereby optimizing the user's voice interaction experience.

[0033] Based on the above embodiment, step 120 includes: Determine speech features of the multiple sound zones based on the frequency spectrum information and phase information of the speech signals of the multiple sound zones, and perform inter-sound zone speech separation on the speech signals of the multiple sound zones based on the speech features of the multiple sound zones; Each of the multiple sound zones is divided into sub-zones to obtain multiple sub-zones under each sound zone, and based on the speech features of the multiple sub-zones, the speech signal of the corresponding sound zone is subjected to intra-zone speech separation.

[0034] Specifically, in step 120, the process of performing inter-sound zone speech separation and intra-sound zone speech separation on the speech signals of the multiple sound zones according to the spectrum information and phase information of the speech signals of the multiple sound zones may specifically include: First, the speech features of the multiple sound zones can be determined based on the spectrum information and phase information of the speech signals of the multiple sound zones. Specifically, the spectrum information and phase information of the speech signals can be directly used as the speech features of the corresponding sound zones, or the spectrum information and phase information can be further encoded, fused, etc., and the processed features can be used as speech features. The embodiment of the present invention does not specifically limit this. Then, the speech features of the multiple sound zones can be used to perform inter-sound zone speech separation on the speech signals of the multiple sound zones to separate the signals of different sound zones in the mixed signal, thereby obtaining accurate separation signals in each sound zone that do not contain signals of other sound zones. The separation signals are the inter-sound zone separation results.

[0035] At the same time, each sound zone can be divided into sub-areas to divide it into multiple sub-areas, thereby obtaining multiple sub-areas under each sound zone. Here, the rules for sub-area division of different sound zones and the number of sub-areas obtained by division can be the same or different. For example, the main driving sound zone and the co-pilot sound zone can be divided into two left and right sub-areas, and the main driving rear sound zone and the co-pilot rear sound zone can be divided into two left and right sub-areas, or divided into three left, middle and right sub-areas. The embodiment of the present invention does not make specific limitations on this.

[0036] After completing the division of the sound zone and obtaining multiple sub-zones under each sound zone, in the embodiment of the present invention, the speech signal of the corresponding sound zone can be subjected to speech separation according to the speech characteristics of each sub-zone, that is, intra-sound zone speech separation is performed to separate the signals of different sub-zones in the sound zone, thereby obtaining separation signals of each sub-zone, which is the intra-sound zone separation result. The speech characteristics of each sub-zone can be determined by the spectrum information and phase information of the speech signal collected by the array elements (array elements in the microphone array) in each sub-zone, and the specific process is basically the same as the determination of the speech characteristics of the sound zone.

[0037] It should be noted that the processes of inter-sound interval speech separation and intra-sound interval speech separation can be realized by the model. That is, the speech features of multiple sound zones can be input into the inter-sound interval separation model, so that the model performs speech separation on the speech signals of the multiple sound zones according to the input speech features, and outputs the inter-sound interval separation results; and the speech features of multiple sub-sound zones can be input into the intra-sound interval separation model, so that the model performs speech separation on the speech signals of the corresponding sound zones according to the input speech features, and outputs the intra-sound interval separation results.

[0038] It is worth noting that before the speech features are input into the model, the embodiment of the present invention also needs to pre-train the inter-sound interval separation model and the intra-sound interval separation model. That is, the inter-sound interval separation model and the intra-sound interval separation model can be pre-trained using sample speech signals and sample separation results corresponding to the sample speech signals.

[0039] Specifically, the sample speech signal used for training may be collected in advance, and the sample speech signal may be a speech signal recorded in a vehicle scene, or a simulated signal, which is not specifically limited in the embodiment of the present invention. If a simulated signal is used as the sample speech, the room impulse response of different areas in the vehicle may be simulated and generated in advance using the Image method, and the room impulse response may be convolved with the clean speech signal, and then a noise signal may be added according to different signal-to-noise ratios to generate a noisy signal, which is the simulated sample speech signal.

[0040] Here, the frequency domain representation of the noisy signal is: In the formula, is a noisy signal, is the signal after convolution processing, is the noise signal, is the frequency subscript, Indicates the time frame.

[0041] When adding a noise signal, the signal-to-noise ratio of the noise signal should cover the actual vehicle noise scenario as much as possible.

[0042] After obtaining the sample speech signal, in the embodiment of the present invention, information extraction can be performed based on the sample speech signals collected by the multiple arrays of the actual recording to obtain the spectrum information and phase information of the sample speech signals of the multiple sample sound zones, and the sample speech features are determined based on this, and then the sample speech features of the multiple sample sound zones are input into the initial sound interval separation model to perform speech separation, and output the predicted sound interval separation results, that is, the speech mask of each sample sound zone. Afterwards, the initial sound interval separation model can be trained based on the sample sound interval separation results and the predicted sound interval separation results, so that it learns the mapping relationship between input and output, and finally a trained sound interval separation model can be obtained.

[0043] The sample sound interval separation result can be calculated by the following formula after determining the sample speech signal: In the formula, is the time-frequency mask of each sample sound area, that is, the sample sound interval separation result, which is calculated by taking the amplitude spectrum of Y and S exist The proportion occupied by is obtained.

[0044] Correspondingly, the training of the intra-range separation model is basically the same as that of the inter-range separation model, and the sample speech signals used for training can also be recorded speech signals or simulated signals. During training, the sample speech features of multiple sample sub-ranges are used as input, and the speech masks (predicted intra-range separation results) of multiple sample sub-ranges are used as output. According to the predicted output and the labels corresponding to the samples, the initial intra-range separation model is trained to obtain the intra-range separation model.

[0045] Here, the loss function used in training the tone interval separation model and the intra-tone separation model is the mean square error loss function, and the learning rate is set to 0.001. The initial tone interval separation model and the initial intra-tone separation model can be constructed based on a convolutional neural network or a recurrent neural network.

[0046] In an embodiment of the present invention, when performing speech separation, the inter-sound zone speech separation is first performed according to the speech characteristics of each sound zone, and then the speech signal of each single array is separated within the sound zone. Finally, the two separation results are fused. Through more detailed area division, more accurate speech separation is achieved, which helps to improve subsequent positioning effects and optimize the interactive experience.

[0047] Based on the above embodiment, based on the spectrum information and phase information of the speech signals of the multiple sound zones, the speech features of the multiple sound zones are determined, including: Performing fixed beam forming on speech signals of multiple sound zones to obtain fixed beams of multiple sound zones; Based on the frequency spectrum information and phase information corresponding to the fixed beams of the multiple sound zones, the speech features of the multiple sound zones are determined.

[0048] Specifically, the process of determining the speech features of the multiple sound zones by using the spectrum information and phase information of the speech signals of the multiple sound zones may specifically include: In order to improve the signal-to-noise ratio and enhance the signal of the speech signal, in an embodiment of the present invention, after acquiring the speech signals of multiple sound zones, the speech signals of each sound zone may be beamformed to increase the proportion of effective components in the signal, thereby obtaining an enhanced signal. Here, specifically, fixed beams may be formed for the speech signals of each sound zone, and fixed beams of each sound zone may be obtained by fixed beamforming, thereby achieving an improvement in the signal-to-noise ratio.

[0049] Furthermore, the spectrum information and phase information of the fixed beam of each sound zone can be solved, that is, after the voice signal of each sound zone is fixed with a beam, the spectrum information of the fixed beam of each sound zone can be calculated. The spectrum information here can be a logarithmic power spectrum; for example, when there are four sound zones on the target vehicle, the logarithmic power spectra of the four sound zones are solved, which are respectively recorded as LPS_fix1, LPS_fix2, LPS_fix3 and LPS_fix4. At the same time, the phase information is calculated, that is, the phase difference between adjacent microphones in each sound zone, which is recorded as IPD (Inter-channel Phase Difference). Afterwards, the speech features of each sound zone can be determined based on this logarithmic power spectrum and phase difference. Preferably, in an embodiment of the present invention, the logarithmic power spectrum and phase difference are directly used as speech features.

[0050] Based on the above embodiment, the inter-sound zone separation result includes the separation results of the multiple sound zones, and the intra-sound zone separation result includes the separation results of the multiple sub-sound zones under the corresponding sound zone; Step 130 includes: From the intra-range separation results of each range, the separation results of the target sub-range are obtained by screening; Based on the separation result of the target consonant area and the separation result of the sound area corresponding to the target consonant area, the speech separation result of the corresponding sound area is determined.

[0051] Specifically, in step 130, the process of determining the speech separation results of multiple sound zones according to the sound zone separation results obtained by the sound zone speech separation and the sound zone separation results obtained by the sound zone speech separation may specifically include: After two speech separations, we can get the separation signals of each zone and the separation signals of each sub-zone under each zone, that is, the separation results of multiple zones and the separation results of multiple sub-zones under each zone; after that, we can merge the two separation results, specifically, we can first take each zone as a unit, select a maximum separation signal from the separation results within each zone, and record it as the separation result of the target sub-zone under each zone. For example, if there are four zones, each of which has three sub-zones, the separation signal of each zone can be obtained by speech separation between zones, which is recorded as ; Speech separation within the sound area can obtain the separation signal of each sub-sound area, which is recorded as ; then for each , which can be obtained from its corresponding three Select the largest one from the above, and record it as , which is the separation result of the target consonant area.

[0052] Then, the separation result of the target sub-sound area can be compared with the separation result of the corresponding sound area to select the largest separation signal as the final separation signal of the corresponding sound area. With the corresponding Compare them and take the maximum value as the final separation signal of the corresponding sound zone. In this way, the final separation signal of each sound zone can be obtained, that is, the speech separation results of multiple sound zones.

[0053] Based on the above embodiment, based on the separation result of the target consonant area and the separation result of the sound area corresponding to the target consonant area, determining the speech separation result of the corresponding sound area includes: Determine a target separation result of the corresponding sound area based on the separation result of the target consonant area and the separation result of the sound area corresponding to the target consonant area; Narrowband enhancement is performed based on the target separation result of the corresponding sound area to obtain the speech separation result of the corresponding sound area.

[0054] Specifically, the process of determining the speech separation result of the corresponding sound area according to the separation result of the target consonant area and the separation result of the sound area corresponding to the target consonant area specifically includes: Considering the poor effect of speech separation in harsh scenarios, the signal obtained by barely separating is also easily distorted, and speech wake-up cannot be effectively performed. In the embodiment of the present invention, in order to further improve the quality of the signal after speech separation, thereby improving the wake-up rate of the backend and optimizing the user experience, the results of the two speech separations are fused here to obtain the final speech separation result. The result fusion can be performed first. The fusion process has been described in detail above and will not be repeated here. Then, the target separation result obtained by fusion is enhanced to enhance the separated signal and obtain the final separation signal, that is, the speech separation result.

[0055] Specifically, the target separation result can be enhanced by adaptive narrowband enhancement, that is, the target separation result of the corresponding sound zone is enhanced by narrowband enhancement, so as to obtain the enhanced target separation result of the corresponding sound zone, which is the speech separation result of the corresponding sound zone. After that, the back-end wake-up and positioning can be performed according to the speech separation result, which can greatly improve the wake-up rate, thereby optimizing the user's voice interaction experience.

[0056] Based on the above embodiment, narrowband enhancement is performed based on the target separation result of the corresponding sound area to obtain the speech separation result of the corresponding sound area, including: Determine a covariance matrix of the corresponding tone area based on the target separation result of the corresponding tone area and the speech signal; Determine a narrowband signal corresponding to the tone range based on a covariance matrix corresponding to the tone range; Based on the narrowband signal of the corresponding sound area and the target separation result, narrowband enhancement is performed to obtain the speech separation result of the corresponding sound area.

[0057] Specifically, the process of performing narrowband enhancement according to the target separation result of the corresponding sound area to obtain the speech separation result of the corresponding sound area may specifically include: First, the parameters of the beamformer for narrowband enhancement can be calculated according to the target separation result of the corresponding sound zone. Specifically, the covariance matrix of the beamformer can be updated using the target separation result. The covariance matrix here includes the speech covariance matrix and the noise covariance matrix.

[0058] Then, the narrowband signal of the corresponding sound zone can be determined according to the updated covariance matrix. Here, the beamformer after parameter update is applied, and the narrowband signal of the corresponding sound zone is derived using the guidance of the target separation result of the corresponding sound zone. The beamformer here can be understood as any adaptive beamforming algorithm.

[0059] After that, the speech separation result of the corresponding sound zone can be obtained according to the narrowband signal of the corresponding sound zone and the target separation result of the corresponding sound zone. Specifically, the narrowband signal of the corresponding sound zone can be obtained after the adaptive narrow beam, and the narrowband signal can be multiplied by the corresponding separation signal to obtain the final separation signal.

[0060] Based on the above embodiment, the speech separation result of the corresponding sound zone can be expressed by the following formula: In the formula, For the The speech separation results of the sound zones are For the The target separation result of each sound zone is For the A narrow wave signal in each sound range.

[0061] The speech covariance matrix and noise covariance matrix are updated as follows: In the formula, is the speech covariance matrix, is the noise covariance matrix, For the The speech signal of the sound zone, is the conjugate transpose.

[0062] In an embodiment of the present invention, a fixed beam is first formed for the speech signal of each sound zone to improve the signal-to-noise ratio, and the enhanced speech signal is used to perform inter-sound zone speech separation and intra-sound zone speech separation. The two separation results are fused, and the target separation result is enhanced using a narrow beam. This can greatly improve the precision and accuracy of speech separation, ensure that the separated signal is not easily distorted, and can effectively wake up the back end, thereby improving the wake-up rate and positioning rate of the back end, greatly optimizing the user's voice interaction experience.

[0063] Based on the above embodiments, Figure 2 It is the overall flow chart of the speech separation method provided by the present invention, such as Figure 2 As shown, the method includes: First, voice signals of multiple sound zones on the target vehicle are acquired.

[0064] Then, fixed beam forming is performed on the speech signals of the multiple sound zones to obtain fixed beams of the multiple sound zones, and speech features of the multiple sound zones are determined based on the frequency spectrum information and phase information corresponding to the fixed beams of the multiple sound zones.

[0065] Subsequently, based on the speech features of the multiple sound zones, the speech signals of the multiple sound zones are subjected to inter-zone speech separation, and each of the multiple sound zones is divided into sub-zones to obtain multiple sub-zones under each sound zone, and based on the speech features of the multiple sub-zones, the speech signals of the corresponding sound zones are subjected to intra-zone speech separation.

[0066] Afterwards, the separation results of the target sub-zone are screened from the intra-zone separation results of each zone; based on the separation results of the target sub-zone and the separation results of the zone corresponding to the target sub-zone, the target separation results of the corresponding zone are determined. The inter-zone separation results include the separation results of multiple zones, and the intra-zone separation results include the separation results of multiple sub-zones under the corresponding zone. Finally, based on the target separation result and speech signal of the corresponding sound zone, the covariance matrix of the corresponding sound zone is determined; based on the covariance matrix of the corresponding sound zone, the narrowband signal of the corresponding sound zone is determined; based on the narrowband signal of the corresponding sound zone and the target separation result, narrowband enhancement is performed to obtain the speech separation result of the corresponding sound zone.

[0067] The method provided by the embodiment of the present invention extracts the spectrum information and phase information of the voice signals of multiple sound zones on the target vehicle, and based on this, performs inter-sound zone voice separation and intra-sound zone voice separation on the voice signals of the multiple sound zones, and determines the voice separation results of the multiple sound zones based on the inter-sound zone separation results obtained by the inter-sound zone voice separation and the intra-sound zone separation results obtained by the intra-sound zone voice separation, thereby overcoming the defects of the traditional scheme that the vehicle-mounted voice separation effect is poor under harsh scenarios, the voice separation and positioning cannot be accurately performed, and the subsequent interaction process is affected. Signal acquisition through multiple microphone arrays can better obtain spatial information, and on this basis, two voice separations with different accuracies are performed, and the results are fused, thereby achieving more accurate voice separation, and being able to obtain accurate and reliable voice separation results, effectively improving the wake-up rate of the vehicle-mounted interactive system and the positioning accuracy, thereby optimizing the user's voice interaction experience.

[0068] The speech separation device provided by the present invention is described below. The speech separation device described below and the speech separation method described above can be referred to each other.

[0069] Figure 3 Schematic diagram of the structure of the speech separation device provided by the present invention. Figure 3 As shown, the device comprises: An acquisition unit 310 is used to acquire voice signals of multiple sound zones on a target vehicle; A separation unit 320, configured to perform inter-sound zone speech separation and intra-sound zone speech separation on the speech signals of the plurality of sound zones based on the spectrum information and phase information of the speech signals of the plurality of sound zones; The fusion unit 330 is used to determine the speech separation results of the multiple sound zones based on the between-sound zone separation results obtained by the between-sound zone speech separation and the within-sound zone separation results obtained by the within-sound zone speech separation.

[0070] The speech separation device provided by the present invention extracts the spectrum information and phase information of the speech signals of multiple sound zones on the target vehicle, and based on this, performs inter-sound zone speech separation and intra-sound zone speech separation on the speech signals of the multiple sound zones, and determines the speech separation results of the multiple sound zones based on the inter-sound zone separation results obtained by the inter-sound zone speech separation and the intra-sound zone separation results obtained by the intra-sound zone speech separation, thereby overcoming the defects of the traditional scheme that the vehicle-mounted speech separation effect is not good under harsh scenes, the speech separation and positioning cannot be performed accurately, and the subsequent interaction process is affected. Signal collection through multiple microphone arrays can better obtain spatial information, and on this basis, two speech separations with different accuracies are performed, and the results are fused, thereby achieving more accurate speech separation, and being able to obtain accurate and reliable speech separation results, effectively improving the wake-up rate of the vehicle-mounted interactive system and the positioning accuracy, thereby optimizing the user's voice interaction experience.

[0071] Based on the above embodiment, the separation unit 320 is used for: Determine speech features of the plurality of sound zones based on the frequency spectrum information and phase information of the speech signals of the plurality of sound zones, and perform inter-speech separation on the speech signals of the plurality of sound zones based on the speech features of the plurality of sound zones; Each of the multiple sound zones is divided into sub-zones to obtain multiple sub-zones under each sound zone, and based on the speech features of the multiple sub-zones, the speech signal of the corresponding sound zone is subjected to intra-zone speech separation.

[0072] Based on the above embodiment, the inter-sound zone separation result includes the separation results of each of the multiple sound zones, and the intra-sound zone separation result includes the separation results of each of the multiple sub-sound zones under the corresponding sound zone; the fusion unit 330 is used to: From the intra-range separation results of each range, the separation results of the target sub-range are obtained by screening; Based on the separation result of the target consonant area and the separation result of the area corresponding to the target consonant area, a speech separation result of the area corresponding to the target consonant area is determined.

[0073] Based on the above embodiment, the fusion unit 330 is used for: Determine a target separation result of the corresponding sound area based on the separation result of the target subtone area and the separation result of the sound area corresponding to the target subtone area; Narrowband enhancement is performed based on the target separation result of the corresponding sound area to obtain the speech separation result of the corresponding sound area.

[0074] Based on the above embodiment, the fusion unit 330 is used for: Determine a covariance matrix of the corresponding tone area based on the target separation result of the corresponding tone area and the speech signal; Determine a narrowband signal corresponding to the tone range based on a covariance matrix corresponding to the tone range; Based on the narrowband signal of the corresponding sound area and the target separation result, narrowband enhancement is performed to obtain the speech separation result of the corresponding sound area.

[0075] Based on the above embodiment, the separation unit 320 is used for: Performing fixed beam forming on the speech signals of the multiple sound zones to obtain fixed beams of the multiple sound zones; Based on the frequency spectrum information and phase information corresponding to the fixed beams of the multiple sound zones, speech features of the multiple sound zones are determined.

[0076] Figure 4 is a structural diagram of the vehicle-mounted interactive system provided by the present invention, such as Figure 4 As shown, the system includes a plurality of microphone arrays 410, a processor 420 and a display 430; the plurality of microphone arrays 410 are respectively arranged in a plurality of sound zones on the target vehicle; The processor 420 is used to obtain voice signals of multiple sound zones collected by the multiple microphone arrays 410 on the target vehicle, and perform inter-sound zone voice separation and intra-sound zone voice separation on the voice signals of the multiple sound zones based on the spectrum information and phase information of the voice signals of the multiple sound zones; determine the voice separation results of the multiple sound zones based on the inter-sound zone separation results obtained by the inter-sound zone voice separation and the intra-sound zone separation results obtained by the intra-sound zone voice separation, perform voice wake-up and voice interaction based on the voice separation results of the multiple sound zones, obtain reply content, and control the display 430 to display the reply content.

[0077] The in-vehicle interactive system provided by the present invention comprises multiple microphone arrays, a processor and a display; the processor is used to obtain voice signals of multiple sound zones collected by multiple microphone arrays on a target vehicle, and perform between-sound zone voice separation and within-sound zone voice separation according to the frequency spectrum information and phase information of the voice signals of the multiple sound zones; based on the between-sound zone separation results obtained by the between-sound zone voice separation and the within-sound zone separation results obtained by the within-sound zone voice separation, the voice separation results of the multiple sound zones are determined, and based on the voice separation results of the multiple sound zones, voice wake-up and voice interaction are performed to obtain reply content, and the display is controlled to display the reply content, thereby overcoming the defects of poor in-vehicle voice separation effect in harsh scenarios in traditional solutions, inability to accurately perform voice separation and positioning, and affecting subsequent interaction processes, achieving more accurate voice separation, being able to obtain accurate and reliable voice separation results, effectively improving the wake-up rate and positioning accuracy of the in-vehicle interactive system, and thus optimizing the user's voice interaction experience.

[0078] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the voice separation method, which includes: acquiring voice signals of multiple sound zones on the target vehicle; performing inter-sound zone voice separation and intra-sound zone voice separation on the voice signals of the multiple sound zones based on the spectrum information and phase information of the voice signals of the multiple sound zones; and determining the voice separation results of the multiple sound zones based on the inter-sound zone separation results obtained by the inter-sound zone voice separation and the intra-sound zone separation results obtained by the intra-sound zone voice separation.

[0079] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0080] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the speech separation method provided by the above-mentioned methods, and the method includes: acquiring speech signals of multiple sound zones on a target vehicle; performing inter-sound zone speech separation and intra-sound zone speech separation on the speech signals of the multiple sound zones based on the frequency spectrum information and phase information of the speech signals of the multiple sound zones; and determining the speech separation results of the multiple sound zones based on the inter-sound zone separation results obtained by the inter-sound zone speech separation and the intra-sound zone separation results obtained by the intra-sound zone speech separation.

[0081] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the speech separation method provided by the above-mentioned methods, the method comprising: acquiring speech signals of multiple sound zones on a target vehicle; performing inter-sound zone speech separation and intra-sound zone speech separation on the speech signals of the multiple sound zones based on the frequency spectrum information and phase information of the speech signals of the multiple sound zones; and determining the speech separation results of the multiple sound zones based on the inter-sound zone separation results obtained by the inter-sound zone speech separation and the intra-sound zone separation results obtained by the intra-sound zone speech separation.

[0082] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0083] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech separation method, characterized in that: include: Acquire voice signals of multiple sound zones on the target vehicle; Based on the frequency spectrum information and phase information of the speech signals of the multiple sound zones, performing inter-sound zone speech separation and intra-sound zone speech separation on the speech signals of the multiple sound zones; Based on the between-tone separation result obtained by the between-tone separation and the within-tone separation result obtained by the within-tone separation, the speech separation results of the plurality of tone zones are determined.

2. The speech separation method according to claim 1, characterized in that: The step of performing inter-sound zone speech separation and intra-sound zone speech separation on the speech signals of the plurality of sound zones based on the spectrum information and phase information of the speech signals of the plurality of sound zones comprises: Determine speech features of the plurality of sound zones based on the frequency spectrum information and phase information of the speech signals of the plurality of sound zones, and perform inter-speech separation on the speech signals of the plurality of sound zones based on the speech features of the plurality of sound zones; Each of the multiple sound zones is divided into sub-zones to obtain multiple sub-zones under each sound zone, and based on the speech features of the multiple sub-zones, the speech signal of the corresponding sound zone is subjected to intra-zone speech separation.

3. The speech separation method according to claim 1, characterized in that: The inter-sound zone separation result includes the separation results of each of the multiple sound zones, and the intra-sound zone separation result includes the separation results of each of the multiple sub-sound zones under the corresponding sound zone; Determining the speech separation results of the plurality of speech zones based on the between-sound zone speech separation result obtained by the within-sound zone speech separation and the within-sound zone speech separation result obtained by the within-sound zone speech separation comprises: From the intra-range separation results of each range, the separation results of the target sub-range are obtained by screening; Based on the separation result of the target consonant area and the separation result of the area corresponding to the target consonant area, a speech separation result of the area corresponding to the target consonant area is determined.

4. The speech separation method according to claim 3, characterized in that: The step of determining the speech separation result of the corresponding sound area based on the separation result of the target consonant area and the separation result of the sound area corresponding to the target consonant area comprises: Determine a target separation result of the corresponding sound area based on the separation result of the target consonant area and the separation result of the sound area corresponding to the target consonant area; Narrowband enhancement is performed based on the target separation result of the corresponding sound area to obtain the speech separation result of the corresponding sound area.

5. The speech separation method according to claim 4, characterized in that: The narrowband enhancement is performed based on the target separation result of the corresponding sound area to obtain the speech separation result of the corresponding sound area, including: Determine a covariance matrix of the corresponding tone area based on the target separation result of the corresponding tone area and the speech signal; Determine a narrowband signal corresponding to the tone range based on a covariance matrix corresponding to the tone range; Based on the narrowband signal of the corresponding sound area and the target separation result, narrowband enhancement is performed to obtain the speech separation result of the corresponding sound area.

6. The speech separation method according to any one of claims 2 to 5, characterized in that: The determining of speech features of the plurality of sound zones based on the spectrum information and phase information of the speech signals of the plurality of sound zones comprises: Performing fixed beam forming on the speech signals of the multiple sound zones to obtain fixed beams of the multiple sound zones; Based on the frequency spectrum information and phase information corresponding to the fixed beams of the multiple sound zones, speech features of the multiple sound zones are determined.

7. A speech separation device, characterized in that: include: An acquisition unit, used for acquiring voice signals of multiple sound zones on a target vehicle; A separation unit, configured to perform inter-sound zone speech separation and intra-sound zone speech separation on the speech signals of the plurality of sound zones based on the spectrum information and phase information of the speech signals of the plurality of sound zones; The fusion unit is used to determine the speech separation results of the multiple sound zones based on the between-sound zone separation results obtained by the between-sound zone speech separation and the within-sound zone separation results obtained by the within-sound zone speech separation.

8. A vehicle-mounted interactive system, characterized in that: It includes a plurality of microphone arrays, a processor and a display; the plurality of microphone arrays are respectively arranged in a plurality of sound zones on a target vehicle; The processor is used to obtain voice signals of multiple sound zones collected by the multiple microphone arrays on the target vehicle, and perform inter-sound zone voice separation and intra-sound zone voice separation on the voice signals of the multiple sound zones based on the spectrum information and phase information of the voice signals of the multiple sound zones; determine the voice separation results of the multiple sound zones based on the inter-sound zone separation results obtained by the inter-sound zone voice separation and the intra-sound zone separation results obtained by the intra-sound zone voice separation, perform voice wake-up and voice interaction based on the voice separation results of the multiple sound zones, obtain reply content, and control the display to display the reply content.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the speech separation method according to any one of claims 1 to 6 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech separation method according to any one of claims 1 to 6 is implemented.

Images