In-vehicle voice processing method, terminal device and storage medium

Abstract

The invention discloses an in-vehicle voice processing method, terminal equipment and a storage medium. The in-vehicle voice processing method comprises the following steps: acquiring a first audio signal acquired by a first microphone and a second audio signal acquired by a second microphone; in response to the fact that the target distance between the first microphone and the second microphone is smaller than a preset threshold value, performing differential beam processing on the first audio signal to obtain a first processing signal, and performing differential beam processing on the second audio signal to obtain a second processing signal; and inputting the first processing signal and the second processing signal into a multi-channel voice separation model to obtain a first separation signal corresponding to the first microphone and a second separation signal corresponding to the second microphone. According to the scheme, the audio signals generated only belonging to the own sound area are separated, and the accuracy of in-vehicle voice interaction is improved.

Description

Technical Field

[0001] The present application relates to the field of voice processing technology, and in particular to a method for processing in-vehicle voice, a terminal device, and a storage medium. Background Art

[0002] With the rapid development of science and technology, voice interaction technology is gradually applied to in-vehicle interconnection scenarios. Users are becoming more and more accustomed to interacting with in-vehicle devices through voice, and the requirements and demands for in-vehicle voice interaction systems are increasing. In order to meet the voice interaction between each user in the car and the in-vehicle devices, the in-vehicle voice interaction system has launched the in-vehicle multi-zone voice interaction service to expand the scope of voice interaction.

[0003] Due to the dense crowds and complex environment in the car, the audio signal collected by the onboard microphone corresponding to a certain audio zone not only contains the voice in the current audio zone, but also may include the voice in other audio zones and the audio played by the vehicle itself, etc. If the audio signal is directly used as a voice command, it is easy to have response errors, resulting in a poor user experience. Summary of the invention

[0004] The present application provides a method for processing in-vehicle voice, a terminal device, and a storage medium.

[0005] A technical solution adopted by the present application is to provide a method for processing in-vehicle voice, wherein at least a first microphone and a second microphone are arranged inside the vehicle, and each microphone corresponds to a sound zone; the method comprises:

[0006] Acquire a first audio signal collected by the first microphone and a second audio signal collected by the second microphone;

[0007] In response to a target distance between the first microphone and the second microphone being less than a preset threshold, performing differential beam processing on the first audio signal to obtain a first processed signal, and performing differential beam processing on the second audio signal to obtain a second processed signal;

[0008] The first processed signal and the second processed signal are input into a multi-channel speech separation model to obtain a first separated signal corresponding to the first microphone and a second separated signal corresponding to the second microphone; wherein the first separated signal does not include audio signals generated by sound zones not corresponding to the first microphone, and the second separated signal does not include audio signals generated by sound zones not corresponding to the second microphone.

[0009] Optionally, a third microphone is further provided inside the vehicle, and the distances between the third microphone and the first microphone and the second microphone are respectively greater than a preset threshold;

[0010] Inputting the first processed signal and the second processed signal into a multi-channel speech separation model to obtain a first separated signal corresponding to the first microphone and a second separated signal corresponding to the second microphone, including:

[0011] The first processed signal, the second processed signal, and the third audio signal collected by the third microphone are input into a multi-channel speech separation model to obtain the speech processing signal generated by each sound zone, and to obtain the first separated signal, the second separated signal and the third separated signal, wherein the third separated signal does not include the audio signal generated by the sound zone not corresponding to the third microphone.

[0012] Optionally, the first processed signal and the second processed signal are input into a multi-channel speech separation model to obtain a first separated signal corresponding to the first microphone and a second separated signal corresponding to the second microphone, including:

[0013] Performing a short-time Fourier transform on the first processed signal to obtain a first frequency domain signal; and performing a short-time Fourier transform on the second processed signal to obtain a second frequency domain signal;

[0014] Inputting the first frequency domain signal and the second frequency domain signal into a multi-channel speech separation model to obtain a first frequency domain mask corresponding to the first frequency domain signal and a second frequency domain mask corresponding to the second frequency domain signal;

[0015] Calculating the product of the first frequency domain signal and the first frequency domain mask to obtain the separated first frequency domain signal; and calculating the product of the second frequency domain signal and the second frequency domain mask to obtain the separated second frequency domain signal;

[0016] An inverse short-time Fourier transform is performed on the separated first frequency domain signal to obtain a first separated signal; and an inverse short-time Fourier transform is performed on the separated second frequency domain signal to obtain a second separated signal.

[0017] Optionally, the multi-channel speech separation model includes an encoder, a time domain convolutional network and a decoder connected in sequence,

[0018] Inputting the first frequency domain signal and the second frequency domain signal into a multi-channel speech separation model to obtain a first frequency domain mask corresponding to the first frequency domain signal and a second frequency domain mask corresponding to the second frequency domain signal, including:

[0019] Inputting the first frequency domain signal to the encoder to obtain a first encoding result, and inputting the second frequency domain signal to the encoder to obtain a second encoding result;

[0020] Inputting the first encoding result into the time domain convolutional network to obtain a first convolution result; and inputting the second encoding result into the time domain convolutional network to obtain a second convolution result;

[0021] The first convolution result is input into a decoder to obtain a first frequency domain mask; and the second convolution result is input into a decoder to obtain a second frequency domain mask.

[0022] Optionally, the first audio signal is obtained by superimposing a first initial audio signal and a first noise signal, and the second audio signal is obtained by superimposing a second initial audio signal and a second noise signal;

[0023] After the steps of performing an inverse short-time Fourier transform on the separated first frequency domain signal to obtain a first separated signal; and performing an inverse short-time Fourier transform on the separated second frequency domain signal to obtain a second separated signal, the method comprises:

[0024] determining a first loss using a first difference between the first separated signal and the first original audio signal; and determining a second loss using a difference between the second separated signal and the second original audio signal;

[0025] Using the first loss and the second loss, determine the total loss;

[0026] The total loss is used to adjust the parameters of the multi-channel speech separation model.

[0027] Optionally, obtaining a first audio signal collected by a first microphone and a second audio signal collected by a second microphone includes:

[0028] Acquire a first reference audio signal, a second reference audio signal, a first reference noise signal, and a second reference noise signal;

[0029] Determine the room impulse response based on vehicle dimensions, microphone placement, and passenger positions;

[0030] Calculating a convolution result of a first reference audio signal and a room impulse response to obtain a first initial audio signal; and calculating a convolution result of a second reference audio signal and a room impulse response to obtain a second initial audio signal;

[0031] Calculating a convolution result of a first reference noise signal and a room impulse response to obtain a first noise signal; and calculating a convolution result of a second reference noise signal and a room impulse response to obtain a second noise signal;

[0032] The first initial audio signal is superimposed on the first noise signal to obtain a first audio signal; and the second initial audio signal is superimposed on the second noise signal to obtain a second audio signal.

[0033] Optionally, the first microphone corresponds to the first sound range, and the second microphone corresponds to the second sound range;

[0034] Performing differential beam processing on the first audio signal to obtain a first processed signal, and performing differential beam processing on the second audio signal to obtain a second processed signal, including:

[0035] Acquire a first suppression angle of the first microphone relative to the second sound range, and a second suppression angle of the second microphone relative to the first sound range;

[0036] Using a first suppression angle, differential beam processing is performed on the first audio signal to obtain a first processed signal; and using a second suppression angle, differential beam processing is performed on the second audio signal to obtain a second processed signal.

[0037] Optionally, performing differential beam processing on the first audio signal using the first suppression angle to obtain a first processed signal includes:

[0038] Determining a first filter coefficient according to a first suppression angle, a target distance, and a signal angular frequency;

[0039] Using the first filter coefficient, performing differential beam processing on the first audio signal to obtain a first processed signal;

[0040] Using the second suppression angle, performing differential beam processing on the second audio signal to obtain a second processed signal includes:

[0041] Determining a second filter coefficient according to a second suppression angle, a target distance, and a signal angular frequency;

[0042] The second audio signal is subjected to differential beam processing using the second filter coefficient to obtain a second processed signal.

[0043] Another technical solution adopted by the present application is to provide a terminal device, the terminal device comprising a memory and a processor connected to the memory;

[0044] The memory is used to store program data, and the processor is used to execute the program data to implement the in-vehicle voice processing method as described above.

[0045] Another technical solution adopted by the present application is to provide a computer storage medium, which is used to store program data. When the program data is executed by a computer, it is used to implement the in-vehicle voice processing method as described above.

[0046] The beneficial effects of the present application are: in response to the target distance between the first microphone and the second microphone being less than a preset threshold, beamforming processing is performed on the first audio signal and the second audio signal respectively, and a first processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the first microphone and suppressing the audio signals generated from other sound zones; and a second processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the second microphone and suppressing the sound generated from other sound zones. The first processed signal and the second processed signal are respectively input into a multi-channel speech separation model to obtain a first separated signal and a second separated signal, so as to more effectively filter out the audio signals that do not correspond to the own sound zone. Furthermore, the separated signal obtained by the in-vehicle speech processing method provided in the present application can more accurately perform speech interaction tasks such as speech recognition and voice wake-up, thereby improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 It is a flowchart of an embodiment of a method for processing in-vehicle speech provided by the present application;

[0049] FIG. 2A to FIG. 2E is a schematic diagram of the distribution positions of vehicle-mounted microphones under different embodiments provided by the present application;

[0050] Figure 3 It is a flowchart of another embodiment of the method for processing in-vehicle speech provided by the present application;

[0051] Figure 4 It is a structural schematic diagram of an embodiment of a multi-channel speech separation model;

[0052] Figure 5 It is a structural diagram of an embodiment of a terminal device provided by the present application;

[0053] Figure 6 It is a structural diagram of an embodiment of a computer storage medium provided by the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] In order to improve the riding experience of passengers, smart vehicles are equipped with onboard microphones at different locations inside the vehicle, and personalized services are provided to different passengers by acquiring voice control commands from passengers at different locations. However, due to the dense crowds and complex environment inside the vehicle, the voice signals collected by the onboard microphones include not only the target passenger’s voice, but also interference signals from passengers at other locations, audio signals played by the vehicle itself, and various environmental noises. How to accurately perceive the voice signals of passengers at different locations is an urgent problem to be solved.

[0056] This application mainly designs a set of methods to improve the accuracy of separating audio signals. Unlike traditional methods, this application starts from the positional relationship between vehicle-mounted microphones and combines a multi-channel speech separation model to separate speech signals. It can separate the audio signals generated by different sound zones from the signals collected by the vehicle-mounted microphones and mixed with audio generated by different sound zones, and accurately identify users speaking in different sound zones.

[0057] Please refer to Figure 1 , Figure 1 It is a flow chart of an embodiment of a method for processing in-vehicle speech provided in the present application.

[0058] like Figure 1 As shown, the method for processing in-vehicle speech in the embodiment of the present application may specifically include the following steps:

[0059] S1, obtaining a first audio signal collected by a first microphone and a second audio signal collected by a second microphone.

[0060] Among them, at least a first microphone and a second microphone are arranged inside the vehicle, and each microphone corresponds to a sound zone.

[0061] like Figure 2A As shown, mic-1 (microphone 1) and mic-2 (microphone 2) correspond to the first sound zone and the second sound zone respectively, wherein mic-1 and mic-2 form a linear array microphone array.

[0062] like Figure 2BAs shown, mic-1, mic-2, mic-3, and mic-4 correspond to the first sound zone, the second sound zone, the third sound zone, and the fourth sound zone, respectively, wherein mic-1 and mic-2 constitute a linear array microphone array, and mic-3 and mic-4 constitute a linear array microphone array.

[0063] like Figure 2C As shown, Figure 2C This is a schematic diagram of an embodiment of a distributed microphone array layout, where mic-1, mic-2, mic-3, and mic-4 correspond to the first sound zone, the second sound zone, the third sound zone, and the fourth sound zone, respectively.

[0064] like Figure 2D As shown, mic-1, mic-2, mic-3, and mic-4 correspond to the first sound zone, the second sound zone, the third sound zone, and the fourth sound zone, respectively, wherein mic-1 and mic-2 form a linear array microphone array.

[0065] like Figure 2E As shown, mic-1, mic-2, mic-3, mic-4, mic-5, and mic-6 correspond to the first sound zone, the second sound zone, the third sound zone, the fourth sound zone, the fifth sound zone, and the sixth sound zone, respectively, wherein mic-1 and mic-2 constitute a linear array microphone array.

[0066] Figure 2A-2E In the given embodiments, the number of sound zones is an even number. In some possible application scenarios, the number of sound zones may also be an odd number, as long as the number of sound zones in the car is greater than 1.

[0067] It should be understood that the in-vehicle speech processing method provided in this application can be used not only for Figure 2A-2E The microphone arrangement shown in the figure can also be used in vehicles with similar microphone arrangements.

[0068] The method for processing in-vehicle speech provided in the present application is mainly performed by a speech processing device. In some embodiments, the speech processing device may be the vehicle-mounted microphone itself. In some embodiments, the distance measuring device may be a device that is communicatively connected to the vehicle-mounted microphone. For example, the device may be a device for monitoring images, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, and an autonomous vehicle, a robot, a security system, or any one or more products of glasses and helmets for augmented reality or virtual reality. In some possible implementations, the method for processing in-vehicle speech may be implemented by a processor calling computer-readable instructions stored in a memory.

[0069] Specifically, the speech processing device obtains a first audio signal collected by a first microphone, and obtains a second audio signal collected by a second microphone.

[0070] Exemplarily, the first microphone corresponds to the first sound zone, and the second microphone corresponds to the second sound zone. Due to the limited space in the car, the first audio signal inevitably contains the sound produced by the second sound zone (or other sounds that do not belong to the first sound zone). Similarly, the second audio signal inevitably contains the sound produced by the first sound zone (or other sounds that do not belong to the second sound zone). Therefore, the main goal of the present application is to perform a correlation separation operation on the first audio signal, filter out the audio signals that do not belong to the first sound zone in the first audio signal, and finally obtain the first separated signal. Similarly, the present application also performs a correlation separation operation on the second audio signal, filters out the audio signals that do not belong to the second sound zone in the second audio signal, and finally obtains the second separated signal.

[0071] S2, in response to a target distance between the first microphone and the second microphone being less than a preset threshold, performing differential beam processing on the first audio signal to obtain a first processed signal, and performing differential beam processing on the second audio signal to obtain a second processed signal.

[0072] Specifically, the condition for two vehicle-mounted microphones to form a linear microphone array is that the relative distance between the two vehicle-mounted microphones is less than a preset threshold.

[0073] Similarly, when the relative distance between two vehicle-mounted microphones is greater than or equal to a preset threshold, the two vehicle-mounted microphones do not constitute a linear array microphone array. In some embodiments, the preset threshold is less than 50 cm. In some embodiments, the preset threshold is less than 40 cm. In some embodiments, the preset threshold is less than 30 cm. In some embodiments, the preset threshold is less than 20 cm. In some embodiments, the preset threshold is less than 10 cm. Exemplarily, the preset threshold is 15 cm.

[0074] In this embodiment, since the target distance between the first microphone and the second microphone is less than the preset threshold, the first microphone is more likely to obtain the audio of the second sound range, and the second microphone is more likely to obtain the audio of the first sound range. In other words, the audio processing device may have difficulty distinguishing the audio signal that does not belong to the first sound range in the first audio signal, and / or, it may be difficult to distinguish the audio signal that does not belong to the second sound range in the second audio signal.

[0075] In the related technology, beamforming is mainly used. However, the application scenario corresponding to this technology is that the first microphone and the second microphone constitute an additive microphone array, and the additive microphone array determines the shape of its main lobe by setting the main lobe itself. For the first microphone and the second microphone belonging to the vehicle-mounted microphone array, the distance between the microphones is small, and the width of the main lobe of the beam is large, resulting in poor directivity, and the array gain is not obvious. The enhancement effect on the target direction is weak, and the suppression effect on the interference direction is poor.

[0076] The differential beamforming algorithm is a sound source localization and sound separation technology based on microphone arrays, which reflects the difference in spatial sound pressure. Compared with related technologies, the differential microphone array determines the shape of its main lobe by setting the null point direction, and by placing the null point in the interference direction, the speech energy ratio in the target direction and the interference direction is increased. The sensitivity of the array to different incident directions is represented by the beam pattern, and the differential beam satisfies the following relationship:

[0077] B[h(ω),θ]=d G (ω,cosθ)h(ω)

[0078] Where d(ω, cosθ) is the steering vector of the array, ω is the signal angular frequency, θ is the incident direction, h(ω) is the output weight of each microphone, and H is the conjugate transpose. Taking the dual microphone end-fire direction as 0° and the suppression direction as 180° as an example, the beam pattern calculation formula for differential beamforming is:

[0079]

[0080] Where h′(ω) is τ is the ratio of the microphone array spacing to the speed of sound.

[0081] In this embodiment, the speech processing device applies the differential beam algorithm to the audio enhancement of the real vehicle linear array microphone array. For example, the speech processing device sets the zero point to the angle of the second sound zone relative to the microphone array, thereby effectively suppressing the audio energy in the direction of the second sound zone, thereby highlighting the audio signal of the first sound zone, and enhancing the energy difference of the audio signals generated by the first sound zone and the second sound zone.

[0082] For another example, the speech processing device sets the zero point to the angle of the first sound zone relative to the microphone array, thereby effectively suppressing the audio energy in the direction of the first sound zone, thereby highlighting the audio signal of the second sound zone.

[0083] In this embodiment, the first audio signal and the second audio signal are processed by the differential beam algorithm, and the energy difference between the obtained first processed signal and the second processed signal becomes larger, and the signal energy distribution is similar to the sound pickup result of a distributed microphone. A distributed microphone refers to two microphones whose spacing is greater than or equal to a preset threshold.

[0084] S3, inputting the first processed signal and the second processed signal into a multi-channel speech separation model to obtain a first separated signal corresponding to the first microphone and a second separated signal corresponding to the second microphone.

[0085] The first separated signal does not include the audio signal generated by the sound area not corresponding to the first microphone, and the second separated signal does not include the audio signal generated by the sound area not corresponding to the second microphone.

[0086] Specifically, the multi-channel speech separation model uses audio signals acquired by multiple on-board microphones (such as the first microphone and the second microphone) to distinguish and extract independent sound sources in mixed speech. In a noisy environment, it can accurately separate and recognize speech and distinguish different speaking objects. It has good adaptability to different acoustic environments and noise conditions.

[0087] In the above scheme, in response to the target distance between the first microphone and the second microphone being less than a preset threshold, the first audio signal and the second audio signal are respectively subjected to beamforming processing, and the first processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the first microphone and suppressing the audio signals generated from other sound zones; and the second processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the second microphone and suppressing the sound generated from other sound zones. The first processed signal and the second processed signal are respectively input into the multi-channel speech separation model to obtain the first separated signal and the second separated signal, so as to more effectively filter out the audio signals that do not correspond to the own sound zone. Furthermore, the separated signal obtained by the in-vehicle speech processing method provided in the present application can more accurately perform speech interaction tasks such as speech recognition and speech wake-up, thereby improving the user experience.

[0088] Another embodiment of the in-vehicle speech processing method provided by the present application may specifically include the following steps:

[0089] S11, obtaining a first audio signal collected by a first microphone and a second audio signal collected by a second microphone.

[0090] Among them, a third microphone is also arranged inside the vehicle, and the distance between the third microphone and the first microphone and the second microphone is respectively greater than a preset threshold.

[0091] S12, in response to the target distance between the first microphone and the second microphone being less than a preset threshold, performing differential beam processing on the first audio signal to obtain a first processed signal, and performing differential beam processing on the second audio signal to obtain a second processed signal.

[0092] S13, input the first processed signal, the second processed signal, and the third audio signal collected by the third microphone into a multi-channel speech separation model to obtain the speech processing signal generated by each sound zone, and obtain the first separated signal, the second separated signal and the third separated signal.

[0093] The third separated signal does not include the audio signal generated by the sound area not corresponding to the third microphone.

[0094] In this embodiment, since the distance between the third microphone and the first microphone is greater than the preset threshold, the distance between the third microphone and the second microphone is greater than the preset threshold, that is, the third microphone does not form a linear array microphone array with the first microphone, and the third microphone does not form a linear array microphone array with the second microphone.

[0095] Furthermore, since the third microphone does not form a linear microphone array with any of the first microphone and the second microphone, the speech processing device can more easily perform corresponding enhancement operations on the third audio signal. Therefore, the speech processing device does not need to perform corresponding differential beam processing on the third audio signal collected by the third microphone, and can directly input the third audio signal into the multi-channel speech separation model.

[0096] It should be noted that the third microphone and the first microphone constitute a distributed microphone, and the third microphone and the second microphone constitute a distributed microphone. Therefore, there is a relatively obvious energy difference between the third audio signal and the first processed signal, and there is a relatively obvious energy difference between the third audio signal and the second processed signal. Therefore, there is no need to perform corresponding differential beam processing on the third audio signal.

[0097] In the above scheme, in response to the target distance between the first microphone and the second microphone being less than a preset threshold, the first audio signal and the second audio signal are respectively subjected to beamforming processing, and the first processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the first microphone and suppressing the audio signals generated from other sound zones; and the second processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the second microphone and suppressing the sound generated from other sound zones. The first processed signal and the second processed signal are respectively input into the multi-channel speech separation model to obtain the first separated signal and the second separated signal, so as to more effectively filter out the audio signals that do not correspond to the own sound zone. Furthermore, the separated signal obtained by the in-vehicle speech processing method provided in the present application can more accurately perform speech interaction tasks such as speech recognition and speech wake-up, thereby improving the user experience.

[0098] Another embodiment of the in-vehicle speech processing method provided by the present application may specifically include the following steps:

[0099] S21, obtaining a first audio signal collected by a first microphone and a second audio signal collected by a second microphone.

[0100] Among them, at least a first microphone and a second microphone are arranged inside the vehicle, and each microphone corresponds to a sound zone.

[0101] S22, in response to the target distance between the first microphone and the second microphone being less than a preset threshold, performing differential beam processing on the first audio signal to obtain a first processed signal, and performing differential beam processing on the second audio signal to obtain a second processed signal.

[0102] S23, performing short-time Fourier transform on the first processed signal to obtain a first frequency domain signal; and performing short-time Fourier transform on the second processed signal to obtain a second frequency domain signal.

[0103] Specifically, the speech processing device performs short-time Fourier transform (STFT) on the first processed signal belonging to the time domain signal to obtain a first frequency domain signal. In addition, the speech processing device performs short-time Fourier transform (STFT) on the second processed signal belonging to the time domain signal to obtain a second frequency domain signal.

[0104] This step realizes the conversion of the time domain signal into the frequency domain signal, so that in the subsequent steps, the multi-channel speech separation model can extract the information of the audio signal generated by the sound zone according to the frequency domain characteristics of the signal.

[0105] S24, inputting the first frequency domain signal and the second frequency domain signal into a multi-channel speech separation model to obtain a first frequency domain mask corresponding to the first frequency domain signal and a second frequency domain mask corresponding to the second frequency domain signal.

[0106] Specifically, the speech processing device inputs the first frequency domain signal into the multi-channel speech separation model to obtain a first frequency domain mask corresponding to the first frequency domain signal, and inputs the second frequency domain signal into the multi-channel speech separation model to obtain a second frequency domain mask corresponding to the second frequency domain signal.

[0107] In some embodiments, the first microphone corresponds to the first sound zone, and the second microphone corresponds to the second sound zone. Further, the first frequency domain mask corresponds to the mask of the first sound zone, and the second frequency domain mask corresponds to the mask of the second sound zone.

[0108] S25, calculating the product of the first frequency domain signal and the first frequency domain mask to obtain the separated first frequency domain signal; and calculating the product of the second frequency domain signal and the second frequency domain mask to obtain the separated second frequency domain signal.

[0109] In some embodiments, the speech processing device calculates the product of the first frequency domain signal and the first frequency domain mask, and uses the product result as the separated first frequency domain signal.

[0110] In some embodiments, the speech processing device calculates the product of the second frequency domain signal and the second frequency domain mask, and uses the product result as the separated second frequency domain signal.

[0111] S26, performing an inverse short-time Fourier transform on the separated first frequency domain signal to obtain a first separated signal; and performing an inverse short-time Fourier transform on the separated second frequency domain signal to obtain a second separated signal.

[0112] It should be noted that the signal output by S25 is a frequency domain signal and cannot be directly used as an input signal for subsequent speech recognition and voice wake-up. Therefore, the signal output by S25 needs to be converted from a frequency domain signal to a time domain signal.

[0113] In some embodiments, the first microphone corresponds to the first sound zone, and the second microphone corresponds to the second sound zone. For example, if the first sound zone does not generate sound (for example, no one is speaking in the first sound zone), the first separation signal is a silent signal. For another example, if the first sound zone generates sound (for example, someone is speaking in the first sound zone), the first separation signal is the sound generated by the first sound zone.

[0114] In the above scheme, in response to the target distance between the first microphone and the second microphone being less than a preset threshold, the first audio signal and the second audio signal are respectively subjected to beamforming processing, and the first processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the first microphone and suppressing the audio signals generated from other sound zones; and the second processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the second microphone and suppressing the sound generated from other sound zones. The first processed signal and the second processed signal are respectively input into the multi-channel speech separation model to obtain the first separated signal and the second separated signal, so as to more effectively filter out the audio signals that do not correspond to the own sound zone. Furthermore, the separated signal obtained by the in-vehicle speech processing method provided in the present application can more accurately perform speech interaction tasks such as speech recognition and speech wake-up, thereby improving the user experience.

[0115] In addition, the in-car speech processing method provided by the present application, based on the preprocessing of the microphone linear array, uses a multi-channel speech separation network to further learn the energy differences of audio signals in different channels and different sound zones, and more accurately separates the speech signals generated by each sound zone.

[0116] Another embodiment of the in-vehicle speech processing method provided by the present application may specifically include the following steps:

[0117] S31, acquiring a first audio signal collected by a first microphone and a second audio signal collected by a second microphone.

[0118] Among them, at least a first microphone and a second microphone are arranged inside the vehicle, and each microphone corresponds to a sound zone.

[0119] S32, in response to the target distance between the first microphone and the second microphone being less than a preset threshold, performing differential beam processing on the first audio signal to obtain a first processed signal, and performing differential beam processing on the second audio signal to obtain a second processed signal.

[0120] S33, performing short-time Fourier transform on the first processed signal to obtain a first frequency domain signal; and the speech processing device performing short-time Fourier transform on the second processed signal to obtain a second frequency domain signal.

[0121] Among them, the multi-channel speech separation model includes an encoder, a time domain convolutional network and a decoder connected in sequence.

[0122] S34, inputting the first frequency domain signal into the encoder to obtain a first encoding result, and inputting the second frequency domain signal into the encoder to obtain a second encoding result.

[0123] Specifically, the speech processing device inputs the first frequency domain signal into the encoder to obtain a first encoding result. Also, the speech processing device inputs the second frequency domain signal into the encoder to obtain a second encoding result.

[0124] S35, inputting the first encoding result into the time domain convolutional network to obtain a first convolution result; and the speech processing device inputting the second encoding result into the time domain convolutional network to obtain a second convolution result.

[0125] The temporal convolutional network (TCN) adopted in this embodiment utilizes the characteristics of convolutional neural networks to model long-term dependencies in a shorter time and has better parallel computing capabilities.

[0126] Specifically, the speech processing device inputs the first encoding result into the time domain convolutional network to obtain the first convolution result. And, the speech processing device inputs the second encoding result into the time domain convolutional network to obtain the second convolution result.

[0127] S36, inputting the first convolution result into a decoder to obtain a first frequency domain mask; and, inputting the second convolution result into a decoder to obtain a second frequency domain mask.

[0128] Specifically, the speech processing device inputs the first convolution result into a decoder to obtain a first frequency domain mask. And, the speech processing device inputs the second convolution result into a decoder to obtain a second frequency domain mask.

[0129] S37, calculating the product of the first frequency domain signal and the first frequency domain mask to obtain the separated first frequency domain signal; and calculating the product of the second frequency domain signal and the second frequency domain mask to obtain the separated second frequency domain signal.

[0130] S38, performing an inverse short-time Fourier transform on the separated first frequency domain signal to obtain a first separated signal; and performing an inverse short-time Fourier transform on the separated second frequency domain signal to obtain a second separated signal.

[0131] In the above scheme, in response to the target distance between the first microphone and the second microphone being less than a preset threshold, the first audio signal and the second audio signal are respectively subjected to beamforming processing, and the first processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the first microphone and suppressing the audio signals generated from other sound zones; and the second processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the second microphone and suppressing the sound generated from other sound zones. The first processed signal and the second processed signal are respectively input into the multi-channel speech separation model to obtain the first separated signal and the second separated signal, so as to more effectively filter out the audio signals that do not correspond to the own sound zone. Furthermore, the separated signal obtained by the in-vehicle speech processing method provided in the present application can more accurately perform speech interaction tasks such as speech recognition and speech wake-up, thereby improving the user experience.

[0132] In addition, the in-car speech processing method provided by the present application, based on the preprocessing of the microphone linear array, uses a multi-channel speech separation network to further learn the energy differences of audio signals in different channels and different sound zones, and more accurately separates the speech signals generated by each sound zone.

[0133] Another embodiment of the in-vehicle speech processing method provided by the present application may specifically include the following steps:

[0134] S41, acquiring a first audio signal collected by a first microphone and a second audio signal collected by a second microphone.

[0135] Among them, at least a first microphone and a second microphone are arranged inside the vehicle, and each microphone corresponds to a sound zone.

[0136] The first audio signal is obtained by superimposing a first initial audio signal and a first noise signal, and the second audio signal is obtained by superimposing a second initial audio signal and a second noise signal;

[0137] In some embodiments, the first initial audio signal is a clean audio signal generated only in the first sound zone inside the vehicle, that is, the first initial audio signal does not contain a noise signal or an audio signal generated in other sound zones. Exemplarily, the first initial audio signal is a speech signal.

[0138] Likewise, the second initial audio signal is a clean audio signal generated only in the second sound zone inside the vehicle.

[0139] In some embodiments, the first noise signal may include an audio signal not generated by the first sound zone, and may also include environmental noise. Similarly, the second noise signal may include an audio signal not generated by the second sound zone, and may also include environmental noise.

[0140] S42, in response to the target distance between the first microphone and the second microphone being less than a preset threshold, performing differential beam processing on the first audio signal to obtain a first processed signal, and performing differential beam processing on the second audio signal to obtain a second processed signal.

[0141] S43, performing short-time Fourier transform on the first processed signal to obtain a first frequency domain signal; and performing short-time Fourier transform on the second processed signal to obtain a second frequency domain signal.

[0142] S44, inputting the first frequency domain signal and the second frequency domain signal into a multi-channel speech separation model to obtain a first frequency domain mask corresponding to the first frequency domain signal and a second frequency domain mask corresponding to the second frequency domain signal.

[0143] S45, calculating the product of the first frequency domain signal and the first frequency domain mask to obtain the separated first frequency domain signal; and calculating the product of the second frequency domain signal and the second frequency domain mask to obtain the separated second frequency domain signal.

[0144] S46, performing an inverse short-time Fourier transform on the separated first frequency domain signal to obtain a first separated signal; and performing an inverse short-time Fourier transform on the separated second frequency domain signal to obtain a second separated signal.

[0145] S47, determining a first loss using a first difference between the first separated signal and the first initial audio signal; and determining a second loss using a difference between the second separated signal and the second initial audio signal.

[0146] It can be understood that, in the training process of the multi-channel speech separation model, the smaller the difference between the first separated signal and the first initial audio signal, the better, and the smaller the difference between the second separated signal and the second initial audio signal, the better.

[0147] Exemplarily, the first difference is determined by a first similarity between the first separated signal and the first initial audio signal. For example, the higher the first similarity, the smaller the first difference, and further, the smaller the first loss. Similarly, the second difference is determined by a second similarity between the second separated signal and the second initial audio signal. The higher the second similarity, the smaller the second difference, and further, the smaller the second loss.

[0148] S48, determining the total loss using the first loss and the second loss.

[0149] Specifically, the relationship between the total loss and the first loss and the second loss satisfies the following relationship:

[0150] Loss_t=Loss1*w1+Loss2*w2

[0151] Among them, Loss_t is the total loss, Loss1 is the first loss, w1 is the first weight, Loss2 is the second loss, and w2 is the second weight.

[0152] S49, using the total loss, to adjust the parameters of the multi-channel speech separation model.

[0153] Specifically, the speech processing device performs single or multiple parameter adjustments on the multi-channel speech separation model so that the total loss is less than a preset loss threshold, thereby completing the training of the multi-channel speech separation model.

[0154] In some possible embodiments, the speech processing device performs operations such as pruning and knowledge distillation on the trained multi-channel speech separation model to achieve lightweight model.

[0155] In the above scheme, in response to the target distance between the first microphone and the second microphone being less than a preset threshold, the first audio signal and the second audio signal are respectively subjected to beamforming processing, and the first processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the first microphone and suppressing the audio signals generated from other sound zones; and the second processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the second microphone and suppressing the sound generated from other sound zones. The first processed signal and the second processed signal are respectively input into the multi-channel speech separation model to obtain the first separated signal and the second separated signal, so as to more effectively filter out the audio signals that do not correspond to the own sound zone. Furthermore, the separated signal obtained by the in-vehicle speech processing method provided in the present application can more accurately perform speech interaction tasks such as speech recognition and speech wake-up, thereby improving the user experience.

[0156] In addition, the in-car speech processing method provided by the present application, based on the preprocessing of the microphone linear array, uses a multi-channel speech separation network to further learn the energy differences of audio signals in different channels and different sound zones, and more accurately separates the speech signals generated by each sound zone.

[0157] Another embodiment of the in-vehicle speech processing method provided by the present application may specifically include the following steps:

[0158] S51, acquiring a first reference audio signal, a second reference audio signal, a first reference noise signal, and a second reference noise signal.

[0159] Specifically, the speech processing device obtains a first reference audio signal, a second reference audio signal, a first reference noise signal, and a second reference noise signal from a related audio data set.

[0160] Exemplarily, the audio data set may be DNS-Challenge2, or may be other audio data sets, which are not limited here.

[0161] S52, determining a room impulse response according to the size of the vehicle, the position of the microphone, and the position of the passengers.

[0162] Specifically, the speech processing device calculates the room impulse response (RIR) according to parameters such as the actual internal size of the vehicle, the installation position of the microphone inside the vehicle, and the positions of the driver and passengers.

[0163] S53, calculating a convolution result of the first reference audio signal and the room impulse response to obtain a first initial audio signal; and calculating a convolution result of the second reference audio signal and the room impulse response to obtain a second initial audio signal.

[0164] Specifically, the speech processing device calculates the convolution result of the first reference audio signal and the room impulse response to obtain the first initial audio signal. Also, the speech processing device calculates the convolution result of the second reference audio signal and the room impulse response to obtain the second initial audio signal.

[0165] It should be noted that due to the complex environment inside the vehicle, if the first reference audio signal and the second reference audio signal obtained from the data set are directly used as the data set of the multi-channel speech separation model, the trained model will have poor speech separation effect during the actual vehicle verification. For example, the separated speech will have serious distortion.

[0166] In this embodiment, the initial audio signal obtained by calculating the convolution result of the reference audio signal and the room impulse response can better simulate the clean signal collected in the vehicle, thereby effectively improving the performance of the model.

[0167] S54, calculating the convolution result of the first reference noise signal and the room impulse response to obtain a first noise signal; and calculating the convolution result of the second reference noise signal and the room impulse response to obtain a second noise signal.

[0168] Specifically, the speech processing device calculates the convolution result of the first reference noise signal and the room impulse response to obtain the first noise signal. Also, the speech processing device calculates the convolution result of the second reference noise signal and the room impulse response to obtain the second noise signal.

[0169] S55 , superimposing the first initial audio signal and the first noise signal to obtain a first audio signal; and superimposing the second initial audio signal and the second noise signal to obtain a second audio signal.

[0170] Specifically, the speech processing device superimposes a first initial audio signal and a first noise signal to obtain a first audio signal. Also, the speech processing device superimposes a second initial audio signal and a second noise signal to obtain a second audio signal.

[0171] Furthermore, the speech processing device uses the first audio signal and the second audio signal to establish a training set, a verification set and a test set.

[0172] It should be noted that steps S51 to S55 can more effectively simulate the audio signal collected by the vehicle microphone in an actual vehicle.

[0173] S56, in response to the target distance between the first microphone and the second microphone being less than a preset threshold, performing differential beam processing on the first audio signal to obtain a first processed signal, and performing differential beam processing on the second audio signal to obtain a second processed signal.

[0174] S57, performing short-time Fourier transform on the first processed signal to obtain a first frequency domain signal; and performing short-time Fourier transform on the second processed signal to obtain a second frequency domain signal.

[0175] S58, inputting the first frequency domain signal and the second frequency domain signal into a multi-channel speech separation model to obtain a first frequency domain mask corresponding to the first frequency domain signal and a second frequency domain mask corresponding to the second frequency domain signal.

[0176] S59, calculating the product of the first frequency domain signal and the first frequency domain mask to obtain the separated first frequency domain signal; and calculating the product of the second frequency domain signal and the second frequency domain mask to obtain the separated second frequency domain signal.

[0177] S60, performing an inverse short-time Fourier transform on the separated first frequency domain signal to obtain a first separated signal; and performing an inverse short-time Fourier transform on the separated second frequency domain signal to obtain a second separated signal.

[0178] S61, determining a first loss using a first difference between the first separated signal and the first initial audio signal; and determining a second loss using a difference between the second separated signal and the second initial audio signal.

[0179] S62, determining a total loss using the first loss and the second loss.

[0180] S63, using the total loss, adjusting parameters of the multi-channel speech separation model.

[0181] In the above scheme, in response to the target distance between the first microphone and the second microphone being less than a preset threshold, the first audio signal and the second audio signal are respectively subjected to beamforming processing, and the first processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the first microphone and suppressing the audio signals generated from other sound zones; and the second processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the second microphone and suppressing the sound generated from other sound zones. The first processed signal and the second processed signal are respectively input into the multi-channel speech separation model to obtain the first separated signal and the second separated signal, so as to more effectively filter out the audio signals that do not correspond to the own sound zone. Furthermore, the separated signal obtained by the in-vehicle speech processing method provided in the present application can more accurately perform speech interaction tasks such as speech recognition and speech wake-up, thereby improving the user experience.

[0182] In addition, the in-car speech processing method provided by the present application, based on the preprocessing of the microphone linear array, uses a multi-channel speech separation network to further learn the energy differences of audio signals in different channels and different sound zones, and more accurately separates the speech signals generated by each sound zone.

[0183] In this embodiment, the audio data of the data set is further processed using the room impulse response, so that the processed audio data can better simulate the actual environment inside the car, so that the trained multi-channel speech separation model can separate audio with less distortion in actual applications.

[0184] Another embodiment of the in-vehicle speech processing method provided by the present application may specifically include the following steps:

[0185] S71, obtaining a first audio signal collected by a first microphone and a second audio signal collected by a second microphone.

[0186] Among them, at least a first microphone and a second microphone are arranged inside the vehicle, and each microphone corresponds to a sound zone.

[0187] The first microphone corresponds to the first sound range, and the second microphone corresponds to the second sound range.

[0188] S72, in response to the target distance between the first microphone and the second microphone being less than a preset threshold, obtaining a first suppression angle of the first microphone relative to the second sound range, and a second suppression angle of the second microphone relative to the first sound range.

[0189] Specifically, in response to the target distance between the first microphone and the second microphone being less than a preset threshold, the speech processing device determines that the first microphone and the second microphone constitute a linear array microphone. Further, the speech processing device determines a first suppression angle θ2 of the first microphone relative to the second sound zone and a second suppression angle θ1 of the second microphone relative to the first sound zone from a large amount of pre-acquired sound pickup data of the first microphone and the second microphone collected from real vehicles and theoretical modeling results.

[0190] S73, performing differential beam processing on the first audio signal using the first suppression angle to obtain a first processed signal; and performing differential beam processing on the second audio signal using the second suppression angle to obtain a second processed signal.

[0191] Specifically, the speech processing device performs differential beam processing on the first audio signal using the first suppression angle θ2 to obtain a first processed signal.

[0192] Furthermore, the speech processing device performs differential beam processing on the second audio signal using the second suppression angle θ1 to obtain a second processed signal.

[0193] S74, inputting the first processed signal and the second processed signal into a multi-channel speech separation model to obtain a first separated signal corresponding to the first microphone and a second separated signal corresponding to the second microphone.

[0194] The first separated signal does not include the audio signal generated by the sound area not corresponding to the first microphone, and the second separated signal does not include the audio signal generated by the sound area not corresponding to the second microphone.

[0195] In the above scheme, in response to the target distance between the first microphone and the second microphone being less than a preset threshold, the first audio signal and the second audio signal are respectively subjected to beamforming processing, and the first processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the first microphone and suppressing the audio signals generated from other sound zones; and the second processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the second microphone and suppressing the sound generated from other sound zones. The first processed signal and the second processed signal are respectively input into the multi-channel speech separation model to obtain the first separated signal and the second separated signal, so as to more effectively filter out the audio signals that do not correspond to the own sound zone. Furthermore, the separated signal obtained by the in-vehicle speech processing method provided in the present application can more accurately perform speech interaction tasks such as speech recognition and speech wake-up, thereby improving the user experience.

[0196] In addition, the in-car speech processing method provided in the present application utilizes a vehicle-mounted microphone array to obtain in-car audio signals, determines the sound source angles on both sides of the array, performs spatial filtering through a differential beam algorithm, suppresses audio signals generated by sound zones corresponding to microphones other than its own, and increases the energy difference of the output speech in the corresponding sound zones.

[0197] Another embodiment of the in-vehicle speech processing method provided by the present application may specifically include the following steps:

[0198] S81, acquiring a first audio signal collected by a first microphone and a second audio signal collected by a second microphone.

[0199] Among them, at least a first microphone and a second microphone are arranged inside the vehicle, and each microphone corresponds to a sound zone.

[0200] The first microphone corresponds to the first sound zone, and the second microphone corresponds to the second sound zone.

[0201] S82, in response to the target distance between the first microphone and the second microphone being less than a preset threshold, obtaining a first suppression angle of the first microphone relative to the second sound zone, and a second suppression angle of the second microphone relative to the first sound zone.

[0202] S83, determining a first filter coefficient according to the first suppression angle, the target distance, and the signal angular frequency.

[0203] In some embodiments, the speech processing device calculates the ratio of the target distance to the speed of sound to obtain a target ratio.

[0204] Furthermore, the speech processing device determines a first filter coefficient according to the first suppression angle, the target ratio, and the signal angular frequency.

[0205] Among them, the first filter coefficient satisfies the following relationship:

[0206]

[0207] Wherein, C1 is the first filter coefficient.

[0208] S84: Perform differential beam processing on the first audio signal using the first filter coefficient to obtain a first processed signal.

[0209] Specifically, the speech processing device performs differential beam processing on the first audio signal using the first filter coefficient to obtain a first processed signal.

[0210] S85, determining a second filter coefficient according to the second suppression angle, the target distance, and the signal angular frequency.

[0211] In some embodiments, the speech processing device determines the second filter coefficient according to the second suppression angle, the target coefficient, and the signal angular frequency.

[0212] Among them, the second filter coefficient satisfies the following relationship:

[0213]

[0214] Wherein, C2 is the first filter coefficient.

[0215] S86: Perform differential beam processing on the second audio signal using the second filter coefficient to obtain a second processed signal.

[0216] Specifically, the speech processing device performs differential beam processing on the second audio signal using the second filter coefficient to obtain a second processed signal.

[0217] S87, input the first processed signal and the second processed signal into a multi-channel speech separation model to obtain a first separated signal corresponding to the first microphone and a second separated signal corresponding to the second microphone.

[0218] The first separated signal does not include the audio signal generated by the sound area not corresponding to the first microphone, and the second separated signal does not include the audio signal generated by the sound area not corresponding to the second microphone.

[0219] In the above scheme, in response to the target distance between the first microphone and the second microphone being less than a preset threshold, the first audio signal and the second audio signal are respectively subjected to beamforming processing, and the first processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the first microphone and suppressing the audio signals generated from other sound zones; and the second processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the second microphone and suppressing the sound generated from other sound zones. The first processed signal and the second processed signal are respectively input into the multi-channel speech separation model to obtain the first separated signal and the second separated signal, so as to more effectively filter out the audio signals that do not correspond to the own sound zone. Furthermore, the separated signal obtained by the in-vehicle speech processing method provided in the present application can more accurately perform speech interaction tasks such as speech recognition and speech wake-up, thereby improving the user experience.

[0220] In addition, the in-car speech processing method provided in the present application utilizes a vehicle-mounted microphone array to obtain in-car audio signals, determines the sound source angles on both sides of the array, performs spatial filtering through a differential beam algorithm, suppresses audio signals generated in sound zones other than the microphone itself, and increases the energy difference of the speech output in the corresponding sound zones.

[0221] See also Figure 3 , Figure 3 It is a flowchart of another embodiment of the in-vehicle voice processing method provided by the present application.

[0222] like Figure 3As shown, another embodiment of the in-vehicle speech processing method provided by the present application may specifically include the following steps:

[0223] S101, obtaining microphone input signals corresponding to different sound zones.

[0224] Specifically, the voice processing device obtains a microphone input signal corresponding to each vehicle-mounted microphone, wherein each vehicle-mounted microphone corresponds to a sound zone.

[0225] S102, whether there is a microphone linear array.

[0226] Specifically, the voice processing device determines whether there is a microphone linear array according to the model of the vehicle.

[0227] If the judgment result is yes, jump to S103. If the judgment result is no, jump to S104.

[0228] In this embodiment, the microphone layout inside the vehicle can be as follows: Figure 2A-2E As shown. Among them, Figure 2A and Figure 2B is the microphone layout of the linear microphone array, Figure 2C The microphone layout for distributed microphones, Figure 2D and Figure 2E It is a hybrid microphone layout, that is, it contains both linear microphone arrays and distributed microphones.

[0229] S103, performing differential beam processing on microphone input signals corresponding to the microphone linear array to obtain multiple processed signals.

[0230] Specifically, if there is a linear microphone array, the speech processing device applies a differential beamforming algorithm to the speech signal of the corresponding channel to enhance the target direction and suppress interference from other directions.

[0231] Among them, the differential beam algorithm is a sound source localization and sound separation technology based on a microphone array, which reflects the difference in spatial sound pressure.

[0232] Differential microphone arrays are different from additive microphone arrays, such as conventional beamforming. Additive microphone arrays determine the shape of their main lobe by setting the main lobe itself. For vehicle-mounted microphone arrays, the microphone array spacing is small and the beam main lobe width is large, resulting in poor directivity, insignificant array gain, weak enhancement effect on the target direction, and poor suppression effect on the interference direction. Differential microphone arrays determine the shape of their main lobe by setting the zero point direction, and by placing a zero point in the interference direction, the speech energy ratio between the target direction and the interference direction is increased. The sensitivity of the array to different incident directions is represented by the beam pattern, and the specific formula is as follows:

[0233] B[h(ω),θ]=dG (ω,cosθ)h(ω)

[0234] Where d(ω,cosθ) is the steering vector of the array, ω is the signal angular frequency, θ is the incident direction, h(ω) is the output weight of each microphone, and H is the conjugate transpose. Taking the dual microphone end-fire direction as 0° and the suppression direction as 180° as an example, the beam pattern calculation formula for differential beamforming is:

[0235]

[0236] h′(ω) is τ is the ratio of the microphone array spacing to the speed of sound.

[0237] In this embodiment, the differential beam algorithm is applied to the speech enhancement of the linear array microphone array of a real vehicle. For example, when the microphone linear array is arranged above the central control screen, the zero point is set as the angle of the driver's side relative to the microphone array, so as to effectively suppress the speech energy in the driver's direction, thereby highlighting the co-driver's voice signal and enhancing the difference in speech energy between the driver and co-driver.

[0238] Based on the above differential beamforming algorithm, S103 may specifically include the following steps:

[0239] S1031, collect a large amount of real vehicle linear array microphone array sound pickup data.

[0240] S1032, based on theoretical modeling and actual vehicle data, analyzing the angles of the sound zones on both sides of the microphone relative to the linear array, that is, determining the suppression angles θ2 and θ1 on each side.

[0241] S1033, using the suppression angle, determine the microphone array filter coefficient -e jστcosθ , differential beam spatial filtering is performed on multi-channel speech signals.

[0242] The voice information captured by the distributed microphone array in different sound zones has obvious energy differences, while the energy and phase of the voice signals picked up by different microphones in the linear array microphone array are relatively small due to the small array spacing. The differential beam algorithm is used to spatially filter the multi-channel voice signals of the linear array microphone array, so that the voice output in different sound zones has obvious energy differences, which is conducive to the subsequent multi-channel voice separation model to accurately identify human voices in different positions and improve the quality and accuracy of voice interaction pickup.

[0243] S104, inputting the multiple processed signals and / or other microphone input signals into a multi-channel speech separation model to separate speech signals corresponding to different sound ranges.

[0244] Specifically, the speech processing device inputs the multi-channel signal (processed signal and / or other microphone input signal) through the above steps into the multi-channel speech separation model, and the result output by the model is the speech information corresponding to different sound zones.

[0245] The multi-channel speech separation model uses speech signals captured by multiple microphones to distinguish and extract independent sound sources in mixed speech. In a noisy environment, it can accurately separate and recognize speech and distinguish different speakers. It has good adaptability to different acoustic environments and noise conditions.

[0246] The structural diagram of an embodiment of a multi-channel speech separation model provided in this application is as follows Figure 4 shown.

[0247] like Figure 4 As shown in the figure, the multi-channel speech separation model consists of an encoder, a temporal convolutional network (TCN) and a decoder connected in sequence. TCN can capture long-distance dependencies of sequences, which is very useful in speech-related tasks. For linear array microphone arrays and hybrid distributions, after the differential beam algorithm, the energy difference of signals in different sound zones of the corresponding channels becomes larger, and the signal energy distribution is similar to the sound pickup result of distributed microphones. Therefore, for microphone layouts of different formations, combined with the differential beam algorithm, the multi-channel speech separation model can be used to learn the energy differences of passenger speech signals in different channels and different sound zones, and separate the speech signals in each sound zone.

[0248] In some embodiments, the model training and lightweight process is as follows:

[0249] S1041, calculate the room impulse response (RIR) according to the actual vehicle size, microphone position, passenger position, etc.

[0250] S1042 randomly selects DNS-Challenge2 single-channel clean speech, noise and RIR for convolution to obtain multi-channel speech signals based on real vehicle scenarios as training sets, validation sets and test sets.

[0251] S1043, if the microphone layout includes a linear microphone array, apply differential beamforming algorithm processing to the corresponding channel signal; if not, jump to S1044.

[0252] S1044, converting the multi-channel speech signal into the time-frequency domain, Figure 4The multi-channel separation model shown is trained for speech separation, and the model output is a mask for the corresponding sound zone. The microphone input is multiplied by the mask of each channel to obtain the speech signal of each sound zone. If a passenger speaks, the corresponding passenger voice is output. If no one speaks, the sound zone outputs silence.

[0253] S1045 adjusts network parameters and quantizes the model to obtain a lightweight vehicle-mounted multi-channel speech separation model to achieve accurate recognition of human voices at different locations.

[0254] Based on the above differential beam and lightweight multi-channel speech separation model, the specific process of vehicle-mounted speech multi-zone separation is as follows:

[0255] S1111, each vehicle microphone captures the voice signal in the vehicle, including human voice, noise, echo, etc., and obtains multi-channel voice output s1…s n , n is the total number of channels.

[0256] S1112, determine whether there is a microphone linear array in the vehicle microphone layout. If so, perform differential beam filtering on the corresponding channels. For example, if the two current channels are signals picked up by the microphone linear array, apply the differential beam algorithm to s1 and s2 to obtain x1 and x2; if there is no linear array, directly proceed to S1113.

[0257] S1113, convert x1x2…s n or s1s2…s n The multi-channel data is converted to the time-frequency domain and input into a lightweight multi-channel speech separation model.

[0258] S1114, obtain the separated speech outputs y1…y in different sound zones n The output data can be used for subsequent voice interaction tasks such as speech recognition and voice wake-up.

[0259] In the above scheme, in response to the target distance between the first microphone and the second microphone being less than a preset threshold, the first audio signal and the second audio signal are respectively subjected to beamforming processing, and the first processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the first microphone and suppressing the audio signals generated from other sound zones; and the second processed signal is obtained by enhancing the audio signal generated by the sound zone corresponding to the second microphone and suppressing the sound generated from other sound zones. The first processed signal and the second processed signal are respectively input into the multi-channel speech separation model to obtain the first separated signal and the second separated signal, so as to more effectively filter out the audio signals that do not correspond to the own sound zone. Furthermore, the separated signal obtained by the in-vehicle speech processing method provided in the present application can more accurately perform speech interaction tasks such as speech recognition and speech wake-up, thereby improving the user experience.

[0260] In addition, the in-car speech processing method provided by the present application uses an in-car microphone array to obtain in-car audio signals, determines the sound source angles on both sides of the array, and performs spatial filtering through a differential beam algorithm to suppress the audio signals generated by the sound zones corresponding to the non-self-microphones, and increase the energy difference of the output speech in the corresponding sound zones. Based on the preprocessing of the microphone linear array, a multi-channel speech separation network is used to further learn the energy differences of audio signals in different channels and different sound zones, and more accurately separate the speech signals generated by each sound zone.

[0261] Please continue to see Figure 5 , Figure 5 The terminal device 500 of the embodiment of the present application includes a processor 51 and a memory 52 .

[0262] The processor 51 and the memory 52 are connected to the bus. The memory 52 stores program data. The processor 51 is used to execute the program data to implement the in-vehicle speech processing method described in the above embodiment.

[0263] In the embodiment of the present application, the processor 51 may also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip having the ability to process signals. The processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 51 may also be any conventional processor, etc.

[0264] This application also provides a computer storage medium, please continue to refer to Figure 6 , Figure 6 It is a structural diagram of an embodiment of a computer storage medium provided in the present application. The computer storage medium 600 stores program data 61. When the program data 61 is executed by the processor, it is used to implement the in-vehicle voice processing method of the above embodiment.

[0265] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0266] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Equivalent structures or equivalent process changes made by utilizing the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for processing in-vehicle speech, characterized in that: At least a first microphone and a second microphone are arranged inside the vehicle, and each microphone corresponds to a sound zone. The method includes: Acquire a first audio signal collected by the first microphone and a second audio signal collected by the second microphone; In response to a target distance between the first microphone and the second microphone being less than a preset threshold, performing differential beam processing on the first audio signal to obtain a first processed signal, and performing differential beam processing on the second audio signal to obtain a second processed signal; The first processed signal and the second processed signal are input into a multi-channel speech separation model to obtain a first separated signal corresponding to the first microphone and a second separated signal corresponding to the second microphone; wherein the first separated signal does not include audio signals generated by sound zones not corresponding to the first microphone, and the second separated signal does not include audio signals generated by sound zones not corresponding to the second microphone.

2. The method according to claim 1, characterized in that A third microphone is further provided inside the vehicle, and the distance between the third microphone and the first microphone and the second microphone is respectively greater than the preset threshold; The step of inputting the first processed signal and the second processed signal into a multi-channel speech separation model to obtain a first separated signal corresponding to the first microphone and a second separated signal corresponding to the second microphone includes: The first processed signal, the second processed signal, and the third audio signal collected by the third microphone are input into the multi-channel speech separation model to obtain the speech processing signal generated by each sound zone, and the first separated signal, the second separated signal and the third separated signal are obtained, wherein the third separated signal does not include the audio signal generated by the sound zone not corresponding to the third microphone.

3. The method according to claim 1, characterized in that The step of inputting the first processed signal and the second processed signal into a multi-channel speech separation model to obtain a first separated signal corresponding to the first microphone and a second separated signal corresponding to the second microphone includes: Performing a short-time Fourier transform on the first processed signal to obtain a first frequency domain signal; and performing a short-time Fourier transform on the second processed signal to obtain a second frequency domain signal; Inputting the first frequency domain signal and the second frequency domain signal into the multi-channel speech separation model to obtain a first frequency domain mask corresponding to the first frequency domain signal and a second frequency domain mask corresponding to the second frequency domain signal; Calculating the product of the first frequency domain signal and the first frequency domain mask to obtain a separated first frequency domain signal; and calculating the product of the second frequency domain signal and the second frequency domain mask to obtain a separated second frequency domain signal; Performing an inverse short-time Fourier transform on the separated first frequency domain signal to obtain the first separated signal; and performing an inverse short-time Fourier transform on the separated second frequency domain signal to obtain the second separated signal.

4. The method according to claim 3, characterized in that The multi-channel speech separation model includes an encoder, a time domain convolutional network and a decoder connected in sequence. The step of inputting the first frequency domain signal and the second frequency domain signal into the multi-channel speech separation model to obtain a first frequency domain mask corresponding to the first frequency domain signal and a second frequency domain mask corresponding to the second frequency domain signal includes: Inputting the first frequency domain signal to the encoder to obtain a first encoding result, and inputting the second frequency domain signal to the encoder to obtain a second encoding result; Inputting the first encoding result into the time domain convolutional network to obtain a first convolution result; and inputting the second encoding result into the time domain convolutional network to obtain a second convolution result; The first convolution result is input into the decoder to obtain the first frequency domain mask; and the second convolution result is input into the decoder to obtain the second frequency domain mask.

5. The method according to claim 3, characterized in that: The first audio signal is obtained by superimposing a first initial audio signal and a first noise signal, and the second audio signal is obtained by superimposing a second initial audio signal and a second noise signal; performing an inverse short-time Fourier transform on the separated first frequency domain signal to obtain the first separated signal; And, after the step of performing an inverse short-time Fourier transform on the separated second frequency domain signal to obtain the second separated signal, the method further comprises: determining a first loss using a first difference between the first separated signal and the first original audio signal; and determining a second loss using a difference between the second separated signal and the second original audio signal; determining a total loss using the first loss and the second loss; The total loss is used to adjust parameters of the multi-channel speech separation model.

6. The method according to claim 5, characterized in that The obtaining of the first audio signal collected by the first microphone and the second audio signal collected by the second microphone includes: Acquire a first reference audio signal, a second reference audio signal, a first reference noise signal, and a second reference noise signal; determining a room impulse response based on the dimensions of the vehicle, the location of the microphone, and the location of the passengers; Calculating a convolution result of the first reference audio signal and the room impulse response to obtain the first initial audio signal; and calculating a convolution result of the second reference audio signal and the room impulse response to obtain the second initial audio signal; Calculating a convolution result of the first reference noise signal and the room impulse response to obtain the first noise signal; and calculating a convolution result of the second reference noise signal and the room impulse response to obtain the second noise signal; The first initial audio signal and the first noise signal are superimposed to obtain the first audio signal; and the second initial audio signal and the second noise signal are superimposed to obtain the second audio signal.

7. The method according to claim 1, characterized in that The first microphone corresponds to the first sound range, and the second microphone corresponds to the second sound range; The performing differential beam processing on the first audio signal to obtain a first processed signal, and performing differential beam processing on the second audio signal to obtain a second processed signal, comprises: Acquire a first suppression angle of the first microphone relative to the second sound range, and a second suppression angle of the second microphone relative to the first sound range; The first audio signal is differentially processed by using the first suppression angle to obtain the first processed signal; and the second audio signal is differentially processed by using the second suppression angle to obtain the second processed signal.

8. The method according to claim 7, characterized in that The step of performing differential beam processing on the first audio signal by using the first suppression angle to obtain the first processed signal includes: Determining a first filter coefficient according to the first suppression angle, the target distance, and the signal angular frequency; Using the first filter coefficient, performing differential beam processing on the first audio signal to obtain the first processed signal; The step of performing differential beam processing on the second audio signal by using the second suppression angle to obtain the second processed signal includes: determining a second filter coefficient according to the second suppression angle, the target distance, and the signal angular frequency; The second audio signal is subjected to differential beam processing using the second filter coefficient to obtain the second processed signal.

9. A terminal device, characterized in that: The terminal device includes a processor and a memory connected to the processor, wherein: The memory stores program instructions; The processor is configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The storage medium stores program instructions, and when the program instructions are executed, the method according to any one of claims 1 to 8 is implemented.

Images