Voice enhancement method, device and equipment and voice interaction system

Abstract

The embodiment of the invention provides a voice enhancement method, device and equipment and a voice interaction system. The method comprises the following steps: acquiring a multi-channel voice signal; determining a sub-band mask of the multi-channel voice signal based on a machine learning model; based on the sub-band mask, calculating a beam weight by using an adaptive beam forming algorithm; and performing enhancement processing on the multi-channel voice signal based on the beam weight, and determining an enhanced single-channel voice signal. The method is used for avoiding dependence on sound source localization in the speech enhancement process and improving the speech enhancement effect.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a speech enhancement method, apparatus, device and speech interaction system. Background Technology

[0002] With the increasing demand for Automatic Speech Recognition (ASR), ensuring the accuracy of microphone array pickup has also received widespread attention.

[0003] Currently, existing microphones typically require locating the sound source before amplifying the speech at that location.

[0004] However, the existing method is difficult to adapt to complex real-world environments, resulting in poor speech enhancement and further reducing the accuracy of speech recognition. Summary of the Invention

[0005] This application provides a speech enhancement method, apparatus, device, and voice interaction system to improve speech enhancement effects.

[0006] In a first aspect, embodiments of this application provide a speech enhancement method, including:

[0007] Acquire multi-channel audio signals;

[0008] Based on a machine learning model, the sub-band mask of the multi-channel speech signal is determined;

[0009] Based on the sub-band mask, the beam weight is calculated using an adaptive beamforming algorithm;

[0010] The multi-channel speech signal is enhanced based on the beam weight to determine the enhanced single-channel speech signal.

[0011] In one possible implementation, acquiring the multi-channel voice signal includes:

[0012] Acquire ambient voice signals from the user's environment using at least two primary microphones;

[0013] The user's voice signal is acquired from the other end of the earpiece using at least one second microphone.

[0014] In one possible implementation, after acquiring the multi-channel speech signal, the method further includes...

[0015] The environmental voice signal and the call voice signal are converted into frequency domain signals.

[0016] In one possible implementation, determining the sub-band mask of the multi-channel speech signal based on a machine learning model includes:

[0017] Using the at least two environmental speech signals as input to the machine learning model, the output of the machine learning model is obtained, and the output includes the sub-band mask of the at least two environmental speech signals.

[0018] Based on the energy difference, correlation, and subband masked data of the speech signals from the first microphone and the second microphone, a working mode is determined. The working mode is either a telephone recording mode or a live recording mode. The correlation is used to characterize the similarity between the speech signals from the first microphone and the second microphone.

[0019] If the working mode is the on-site recording mode, then the call voice signal is filtered out;

[0020] If the working mode is telephone recording mode, then the call voice signal is integrated into the output result.

[0021] In one possible implementation, before determining the subband mask of the multi-channel speech signal based on a machine learning model, the method further includes:

[0022] Noise suppression is performed on the first frequency domain signal using a fixed beamforming algorithm. The first frequency domain signal is obtained by frequency domain transformation of the ambient speech signal in the user's environment.

[0023] In one possible implementation, calculating the beam weights using an adaptive beamforming algorithm based on the sub-band mask includes:

[0024] Using the multi-task learning mechanism of the machine learning model, the speech mask and noise mask of the speech signal for each channel are predicted;

[0025] Based on the speech mask and the noise mask, the covariance matrix in the adaptive beamforming algorithm is constructed. The covariance matrix includes the noise covariance matrix and the speech covariance matrix corresponding to the clean speech after the speech mask.

[0026] The beam weights are determined based on the covariance matrix.

[0027] In one possible implementation, the method further includes converting the enhanced single-channel speech signal from the frequency domain to the time domain.

[0028] Secondly, embodiments of this application provide a voice enhancement device, comprising:

[0029] The acquisition module is used to acquire multi-channel voice signals;

[0030] A mask determination module is used to determine the sub-band mask of the multi-channel speech signal based on a machine learning model;

[0031] The weight determination module is used to calculate the beam weights based on the sub-band mask using an adaptive beamforming algorithm.

[0032] The speech enhancement module is used to enhance the multi-channel speech signal based on the beam weights and determine the enhanced single-channel speech signal.

[0033] Thirdly, embodiments of this application provide a voice enhancement device, including: a microphone array and a processor, wherein the microphone array is connected to the processor, the microphone array is used to acquire multi-channel voice signals; and the processor is used to execute computer execution instructions stored in a memory, causing the processor to perform the method described above.

[0034] Fourthly, embodiments of this application provide a voice interaction system, including:

[0035] Microphone array, used to acquire multi-channel voice signals;

[0036] The processor, connected to the microphone array, is used to generate a sub-band mask for the multi-channel speech signal through a machine learning model, and to calculate beam weights based on the sub-band mask using an adaptive beamforming algorithm to enhance the multi-channel speech signal.

[0037] A speech recognition unit, connected to the processor, is used to recognize and process the enhanced single-channel speech signal.

[0038] The speech enhancement method, apparatus, device, and speech interaction system provided in this application estimate the sub-band mask through a machine learning model. Based on this, the covariance matrix is ​​directly optimized during the adaptive beamforming process, which can avoid the dependence on sound source localization and enhance sound source signals from multiple directions, thereby improving the speech clarity in complex scenes and enhancing the speech enhancement effect. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0040] Figure 1 A schematic diagram of the speech enhancement method provided in this application;

[0041] Figure 2 This is a schematic diagram of the dual voice decision process provided in an embodiment of this application;

[0042] Figure 3 A schematic diagram of the weighting process provided in the embodiments of this application;

[0043] Figure 4 This is a schematic diagram of the speech enhancement process provided in an embodiment of this application;

[0044] Figure 5 This is a schematic diagram of the overall process of the speech enhancement method provided in the embodiments of this application;

[0045] Figure 6 A schematic diagram of the speech enhancement device provided in this application;

[0046] Figure 7 A schematic diagram of the structure of the speech enhancement device provided in this application.

[0047] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0048] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0049] Traditional microphone pickup methods are mainly divided into two types: (1) First, estimate the sound source location and the current noise, and then calculate the beamforming weight matrix in real time based on the sound source location. This requires that the sound source location estimation must be accurate. If it is inaccurate, it will not only fail to enhance the speech, but will also suppress the original speech. Moreover, when there are multiple sound sources at the same time, it can only enhance the sound in one direction at the same time, and the sound in other directions will be suppressed. In addition, when the sound source location changes, the convergence speed of the beamforming weight filter calculated in real time is slow. (2) Calculate the weight matrices of multiple directions in advance, and then use the pre-calculated weight matrices according to the speaker's location. However, when calculating the weight matrices of each direction in advance, the noise used is different from the noise level of the real scene. Similarly, there are scenarios where multiple sounds exist at the same time, and the optimal decision cannot be made. In summary, both methods have common disadvantages: ① The current background noise level cannot be accurately estimated in real time; ② The technical solution fails or is ineffective in low signal-to-noise ratio scenarios.

[0050] To address the aforementioned issues, embodiments of this application provide a speech enhancement method, apparatus, device, and voice interaction system. These methods utilize an adaptive beamforming algorithm combined with a machine learning model to estimate the mask and optimize the covariance matrix estimation, thereby significantly improving speech signal quality and recognition accuracy.

[0051] The voice enhancement method provided in this application can be applied to scenarios such as meeting recording, teleconference, distance education, smart home devices, and in-vehicle voice assistants.

[0052] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0053] Figure 1 A schematic diagram of the speech enhancement method provided in this application is shown below. Figure 1 As shown, this method can be applied to a speech enhancement device. Taking the speech enhancement device as the execution subject as an example, the method includes the following steps:

[0054] Step 110: Acquire multi-channel audio signals;

[0055] Step 120: Determine the sub-band mask of the multi-channel speech signal based on the machine learning model;

[0056] Step 130: Calculate beam weights using an adaptive beamforming algorithm based on the subband mask;

[0057] Step 140: Enhance the multi-channel speech signal based on beam weights to determine the enhanced single-channel speech signal.

[0058] In this embodiment, the sub-band mask is estimated by a machine learning model, and the covariance matrix is ​​directly optimized during the adaptive beamforming process. This avoids dependence on sound source localization, enhances sound source signals from multiple directions, improves speech clarity in complex scenes, and enhances speech enhancement effect.

[0059] Regarding step 110 above, multi-channel voice signals can be acquired by multiple different microphones, such as digital microphones and bone conduction microphones. Digital microphones can pick up voice from the surrounding environment, while bone conduction microphones can pick up voice from the other end of the earpiece. For example, in a conference call scenario, when user A and user B are communicating by phone, the bone conduction microphone on user A's phone can collect the voice from user B, while the digital microphone on user A's phone can collect the voice from user A's surrounding environment.

[0060] Regarding step 120, the machine learning model can refer to a Convolutional Recurrent Network (CRN). A CRN is a deep learning model that combines the advantages of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).

[0061] The CRN network structure can include front-end convolutional layers, intermediate recurrent layers, and terminal fully connected layers. The front-end convolutional layers consist of multiple convolutional operations and are used to capture local patterns and hierarchical features in the input data. The intermediate recurrent layers take the convolutionally transformed features as input and perform sequence modeling to understand the relationships between different time steps. The terminal fully connected layers are responsible for the final task decision.

[0062] In this embodiment, the subband mask is a weighted graph for speech separation in the frequency domain, which is used to characterize the proportion of target speech and the proportion of noise at the current time frame T and F frequency points.

[0063] Here, a sub-band can refer to a frequency sub-band. The audio signal is decomposed into multiple frequency points or bands through a Short-Time Fourier Transform (STFT), and a sub-band is equivalent to each frequency point or band. A mask is a value between 0 and 1, representing the probability or intensity ratio of the presence of a certain component (such as target speech) at a certain time-frequency point.

[0064] In this embodiment, CRN takes multi-channel speech as input, processes it through complex convolution and recursive structures, and finally outputs the sub-band mask matrix corresponding to each channel of speech.

[0065] Regarding steps 130 and 140 above, the adaptive beamforming algorithm can refer to the Mask-based Minimum Variance Distortionless Response (MVDR) beamforming algorithm, which optimizes the performance of MVDR by combining sub-band masks to more accurately estimate the target speech or noise.

[0066] In this embodiment, for each channel of speech, the covariance matrix of that channel's speech can be estimated using its corresponding sub-band mask. Then, an MVDR beamformer can be designed, which generates a beamforming weight vector based on the covariance matrix of the channel's speech and the noise covariance matrix (estimated by the noise mask) to enhance the channel's speech.

[0067] In some embodiments, in order to enable adaptive switching of modes, such as when the user is in telephone recording mode, picking up the conversation between the user and the other party, and when the user is in live recording mode (e.g., in a conference scenario), picking up the speech of all speakers in the conference, a first microphone and a second microphone can be configured.

[0068] In this embodiment, environmental voice signals from the user's environment can be acquired from at least two first microphones; and call voice signals from the other end of the user's earpiece can be acquired from at least one second microphone, thereby obtaining multi-channel voice signals.

[0069] The first microphone can be a digital microphone, and a total of four digital microphones can be configured. The second microphone can be a bone conduction microphone, and a total of one bone conduction microphone can be configured. The four digital microphones and one bone conduction microphone can form a 4+1 microphone array.

[0070] Specifically, four digital microphones are used to collect signals from the environment, and one bone conduction microphone is used to collect the sound from the other end of the phone during a call.

[0071] In this embodiment, multiple digital microphones and bone conduction microphones are configured to collect speech from different voice channels. The collaborative work of the bone conduction microphones and digital microphones can effectively extract the other party's speech in telephone recording mode and enhance far-field speech in live recording mode, which can significantly improve the speech quality in low signal-to-noise ratio scenarios. At the same time, it can realize flexible switching between different modes and respond to environmental changes in real time (such as user calls or conference speeches) without recalculating the sound source location or weight matrix, thus improving the system's real-time performance.

[0072] Furthermore, in some embodiments, since the speech signal directly acquired by the microphone is a time-domain signal, although the time-domain signal contains all the information of the sound, it is not convenient for subsequent speech enhancement processing. Therefore, the environmental speech signal and the call speech signal can be converted into a frequency-domain signal.

[0073] Among them, the time-domain signal is the waveform of sound changing over time, with the horizontal axis representing time and the vertical axis representing amplitude (such as the air pressure change caused by sound).

[0074] In this embodiment, the environmental speech signal and the call speech signal are converted into frequency domain signals. The purpose is to convert the continuous waveform signal in the time domain into a two-dimensional matrix representation in the time-frequency domain so that subsequent algorithms can perform fine speech enhancement processing in the frequency domain.

[0075] For example, short-time Fourier transforms can be performed on the environmental speech signals collected by the four digital microphones and the call speech signal collected by the one bone conduction microphone to convert them from the time domain to the frequency domain. At this time, each speech signal becomes a time-spectrum matrix, which contains amplitude and phase information.

[0076] The short-time Fourier transform process can specifically include the following steps:

[0077] 1. Before STFT, the original speech signal is preprocessed, such as pre-emphasis, to boost the high-frequency part and flatten the signal spectrum;

[0078] 2. Select appropriate STFT parameters based on the application scenario, such as framing (each frame is 16 milliseconds long) and windowing (to reduce spectral leakage).

[0079] 4. Perform a short-time Fourier transform. Perform a fast Fourier transform on the windowed signal of each frame to obtain the spectrum of that frame.

[0080] In the embodiments of this application, by converting multiple speech signals to the frequency domain, subsequent algorithms can be processed independently on different frequency sub-bands, thereby achieving a refined speech enhancement effect.

[0081] Additionally, in some embodiments, due to the switching between different modes, to avoid the influence of the call voice signal on the ambient voice signal when using a machine learning model to calculate the sub-band mask, a dual voice decision can be set. Specifically, Figure 2 This is a schematic diagram of the dual voice decision process provided in the embodiments of this application, such as... Figure 2 As shown, it includes the following steps:

[0082] Step 210: Use at least two environmental speech signals as input to the machine learning model and obtain the output of the machine learning model.

[0083] The output includes sub-band masks for at least two ambient speech signals.

[0084] Step 220: Determine the working mode based on the difference and correlation of the voice signal energy between the first microphone and the second microphone, and the data after sub-band masking.

[0085] The operating modes are either telephone recording mode or on-site recording mode, and the correlation is used to characterize the similarity between the voice signal from the first microphone and the voice signal from the second microphone.

[0086] Step 230: If the working mode is on-site recording mode, then filter out the call voice signal.

[0087] Step 240: If the working mode is telephone recording mode, then integrate the call voice signal into the output result.

[0088] In this embodiment, the user is on a call in telephone recording mode. At this time, it is necessary to pick up the content of the call between the user and the other party (i.e., the voice content in the call voice signal). The voice of the other party is collected through the bone conduction microphone mentioned above.

[0089] In live recording mode, when users are having a meeting, they need to capture the voices of all speakers. At this time, the voice signal collected by the bone conduction microphone may contain the user's own voice (because the user's own voice will be conducted through the bones). However, it is not necessary to add it to the final output. Therefore, a decision needs to be made to determine whether to filter it out.

[0090] Specifically, dual-voice decision-making falls into the following two categories:

[0091] Case 1: The voice content of the call voice signal contains voice (and is judged to be the other end's voice, i.e., telephone recording mode) → add the bone conduction signal to the final output.

[0092] Scenario 2: The voice content of the call signal does not contain any voice, or it is only the user's own voice, but the user's own voice is not added by bone conduction in the on-site recording mode → discard the bone conduction signal.

[0093] In this embodiment, the voice signal acquired by the bone conduction microphone can first be preprocessed, for example, using the short-time Fourier transform mentioned above. After preprocessing, the following two-stage decision can be set:

[0094] (1) First judgment: energy and spectral entropy detection.

[0095] The first level of judgment determines the presence of speech by detecting frame energy and spectral entropy. To improve detection accuracy, conditions can be set, such as considering a segment as a genuine speech segment only if speech is detected in multiple consecutive frames.

[0096] (2) Second decision: Use lightweight neural networks for more refined speech classification.

[0097] The second layer of judgment uses local deep learning to classify speech types, distinguishing between peer-to-peer voice calls, environmental noise, and the user's own voice.

[0098] Through the above two-step judgment, it is possible to detect whether the voice content of the call voice signal contains speech, and whether the speech is just the user's own voice.

[0099] In this embodiment of the application, by setting the above-mentioned dual decision, interference from the call voice signal collected by the bone conduction microphone can be avoided in the conference scenario. At the same time, the call voice signal is only integrated into the output result when needed (e.g., in telephone recording mode), which can reduce the computational overhead and achieve energy efficiency optimization.

[0100] Furthermore, in order to improve the speech enhancement effect, after converting the multi-channel speech signal into a frequency domain signal, a fixed beamforming algorithm can be used to suppress noise in the first frequency domain signal.

[0101] The first frequency domain signal is obtained by frequency domain conversion of the environmental speech signal in the user's environment.

[0102] In this embodiment, the fixed beamforming (FBF) algorithm is a spatial filtering-based technique that enhances signals from a specific direction and suppresses noise from other directions by weighting and delaying the frequency domain signals from multiple microphones.

[0103] Taking the four digital microphones mentioned above as an example, the voice signal collected by each digital microphone is converted into a frequency domain signal. By utilizing the spatial relationship of the four digital microphones and through preset fixed weighting coefficients, an auditory spotlight pointing in the direction of the target can be formed.

[0104] In this embodiment, the first frequency domain signal is preprocessed using a fixed beamforming algorithm. While enhancing the target speech, it can slightly suppress noise interference from other directions, providing a relatively clean input signal for the subsequent CRN network. At the same time, it can solve the problem that the subsequent Mask-based MVDR algorithm is sensitive to covariance matrix estimation, which may lead to excessive noise suppression or speech distortion. It balances noise suppression and speech fidelity, and improves system stability.

[0105] In some embodiments, to further improve noise suppression and enhance speech enhancement effectiveness... Figure 3 The weighting process diagram provided for the embodiments of this application is as follows: Figure 3 As shown, it includes the following steps:

[0106] Step 310: Utilize the multi-task learning mechanism of the machine learning model to predict the speech mask and noise mask for each channel of the speech signal;

[0107] Step 320: Construct the covariance matrix in the adaptive beamforming algorithm based on the speech mask and noise mask;

[0108] Step 330: Determine the beam weights based on the covariance matrix.

[0109] The covariance matrix includes the noise covariance matrix and the speech covariance matrix corresponding to the clean speech after speech masking.

[0110] In this embodiment, the machine learning model can be the CRN model mentioned above. A multi-task learning mechanism can be introduced into the CRN model to simultaneously predict both the speech mask and the noise mask. Furthermore, the accuracy of mask estimation can be improved by jointly optimizing the loss function.

[0111] Traditional MVDR beamformers are typically used to enhance sound sources in a specific direction, requiring prior estimation of the sound source location (e.g., a target speaker). In this embodiment, however, a CRN model is used to calculate the speech mask and noise mask for each channel's speech signal. The corresponding speech mask is then used to estimate the covariance matrix of that channel's speech signal. For each channel's speech signal, an MVDR beamformer is designed. This beamformer generates a beamforming weight vector based on the covariance matrix of the channel's speech signal and the noise covariance matrix (estimated from the noise mask) to enhance the sound source. This enables adaptive tracking of different sound sources and enhancement of sound sources in multiple directions.

[0112] In the MVDR beamforming stage, combining speech masking and noise masking to optimize the construction of the covariance matrix can further improve the noise suppression effect.

[0113] In the embodiments of this application, by using a multi-task learning mechanism to jointly optimize speech and noise masks, mask estimation errors can be reduced, mask estimation accuracy can be improved, the construction of the covariance matrix can be further optimized, and noise residue can be reduced.

[0114] Furthermore, in some embodiments, after the speech signals of each channel are enhanced, the enhanced single-channel speech signal can be converted from the frequency domain back to the time domain, thereby finally obtaining the enhanced sound source.

[0115] Specifically, Figure 4 This is a schematic diagram of the speech enhancement process provided in the embodiments of this application, such as... Figure 4 As shown, it includes the following steps:

[0116] Step 410: Short-time Fourier transform.

[0117] Step 420: Fixed beamforming.

[0118] Step 430: Input the convolutional recurrent network.

[0119] Step 440: Calculate the subband mask.

[0120] Step 450: Voice activity detection.

[0121] Step 460: Adaptive beamforming.

[0122] Step 470: Short-time Fourier inverse transform.

[0123] In this embodiment, a 4+1 microphone array is formed by using four digital microphones and one bone conduction microphone. The four microphones are used to collect signals from the environment, and the bone conduction microphone is used to collect the sound from the other end of the phone during a call. The array adaptively switches modes in real time, that is, when the user is on a call, it will pick up the conversation between the user and the other end, and when the user is in a meeting, it will pick up the speech of all speakers in the meeting who speak different dialects.

[0124] Furthermore, Figure 5 This is a schematic diagram of the overall process of the speech enhancement method provided in the embodiments of this application, as shown below. Figure 5 As shown, it specifically includes the following steps:

[0125] Step 510: Perform short-time Fourier transform on the data from the four digital microphones and the data from the one bone conduction channel respectively, converting them from time-domain signals to frequency-domain signals.

[0126] Step 520: Use fixed beamforming to enhance the multiple signals and slightly suppress noise.

[0127] Step 530: Input the signal enhanced by the fixed beamforming module into the convolutional recurrent network model to calculate the subband mask.

[0128] Step 540: Perform dual voice judgment on the bone conduction signal. If there is voice, it is considered to be telephone recording mode, and the bone conduction signal is added to the final output. If it fails the judgment, it is considered to be regular recording, and the bone conduction signal is discarded.

[0129] Step 550: Based on the subband mask calculated by the convolutional recurrent network model, perform adaptive beamforming to ensure that all sound sources at different locations in the environment are effectively enhanced.

[0130] Step 560: Perform a short-time inverse Fourier transform on the final enhanced single-channel frequency domain signal to convert it from the frequency domain to the time domain.

[0131] In this embodiment, Mask-based MVDR technology combined with a machine learning model is used. The CRN neural network model can achieve accurate noise calculation, which greatly improves speech clarity and far-field ASR accuracy. It has good robustness in long-distance and low signal-to-noise ratio scenarios, and does not require estimation of sound source location. It can support scenarios where multiple sound sources exist at the same time, and has advantages such as flexible deployment and low resource consumption.

[0132] Figure 6 A schematic diagram of the speech enhancement device provided in this application is shown below. Figure 6As shown, the speech enhancement device 60 provided in this embodiment includes:

[0133] Acquisition module 601 is used to acquire multi-channel voice signals;

[0134] The mask determination module 602 is used to determine the sub-band mask of the multi-channel speech signal based on a machine learning model;

[0135] The weight determination module 603 is used to calculate beam weights based on sub-band masks using an adaptive beamforming algorithm.

[0136] The speech enhancement module 604 is used to enhance multi-channel speech signals based on beam weights and determine the enhanced single-channel speech signal.

[0137] In one possible implementation, the acquisition module can be specifically used to: acquire ambient voice signals from at least two first microphones; and acquire call voice signals from at least one second microphone.

[0138] In one possible implementation, a frequency domain conversion module is also included to convert ambient speech signals and call speech signals into frequency domain signals.

[0139] In one possible implementation, the mask determination module can specifically be used for:

[0140] Using at least two ambient speech signals as input to a machine learning model, the output of the machine learning model is obtained, and the output includes the subband mask of at least two ambient speech signals.

[0141] Based on the energy difference, correlation, and sub-band masked data of the speech signals from the first and second microphones, the working mode is determined. The working mode is either telephone recording mode or on-site recording mode. The correlation is used to characterize the similarity between the speech signals from the first microphone and the speech signals from the second microphone.

[0142] If the working mode is on-site recording mode, then the call voice signal will be filtered out;

[0143] If the working mode is telephone recording mode, the call voice signal will be integrated into the output result.

[0144] In one possible implementation, a noise suppression module is also included, used to suppress noise in the first frequency domain signal using a fixed beamforming algorithm. The first frequency domain signal is obtained by frequency domain transformation of the ambient speech signal in the user's environment.

[0145] In one possible implementation, the weight determination module can specifically be used for:

[0146] By utilizing the multi-task learning mechanism of machine learning models, the speech mask and noise mask of the speech signal in each channel are predicted.

[0147] Based on speech masking and noise masking, the covariance matrix in the adaptive beamforming algorithm is constructed;

[0148] Beam weights are determined based on the covariance matrix.

[0149] The covariance matrix includes the noise covariance matrix and the speech covariance matrix corresponding to the clean speech after speech masking.

[0150] In one possible implementation, a time-domain conversion module is also included to convert each enhanced speech signal from the frequency domain to the time domain.

[0151] The speech enhancement device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0152] Figure 7 A schematic diagram of the structure of the speech enhancement device provided in this application. Figure 7 As shown, the voice enhancement device 70 provided in this embodiment includes at least one processor 701 and a microphone array 702, wherein the microphone array 702 is connected to the processor 701.

[0153] Optionally, the microphone array includes four digital microphones and one bone conduction microphone.

[0154] In a specific implementation, at least one processor 701 executes computer execution instructions stored in memory, causing at least one processor 701 to perform the above-described method.

[0155] The specific implementation process of processor 701 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0156] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0157] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0158] This application also provides a voice interaction system, including a microphone array, a processor, and a voice recognition unit. The microphone array is connected to the processor. Optionally, the microphone array includes four digital microphones and one bone conduction microphone.

[0159] The microphone array is used to acquire multi-channel speech signals; the processor is used to generate sub-band masks for the multi-channel speech signals through a machine learning model, and calculates beam weights based on the sub-band masks using an adaptive beamforming algorithm to enhance the multi-channel speech signals; the speech recognition unit is connected to the processor and is used to recognize and process the enhanced single-channel speech signals.

[0160] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0161] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0162] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0163] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0164] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0165] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0166] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0168] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A speech enhancement method, characterized in that, include: Acquire multi-channel audio signals; Based on a machine learning model, the sub-band mask of the multi-channel speech signal is determined; Based on the sub-band mask, the beam weight is calculated using an adaptive beamforming algorithm; The multi-channel speech signal is enhanced based on the beam weight to determine the enhanced single-channel speech signal.

2. The method according to claim 1, characterized in that, The acquisition of multi-channel voice signals includes: Acquire ambient voice signals from the user's environment using at least two primary microphones; The user's voice signal is acquired from the other end of the earpiece using at least one second microphone.

3. The method according to claim 2, characterized in that, After acquiring the multi-channel speech signal, the method further includes The environmental voice signal and the call voice signal are converted into frequency domain signals.

4. The method according to claim 2 or 3, characterized in that, The process of determining the sub-band mask of the multi-channel speech signal based on a machine learning model includes: Using the at least two environmental speech signals as input to the machine learning model, the output of the machine learning model is obtained, and the output includes the sub-band mask of the at least two environmental speech signals. Based on at least one of the differences in speech signal energy between the first microphone and the second microphone, correlation, and subband masked data, a working mode is determined, wherein the working mode is either a telephone recording mode or a live recording mode, and the correlation is used to characterize the similarity between the speech signal of the first microphone and the speech signal of the second microphone. If the working mode is the on-site recording mode, then the call voice signal is filtered out; If the working mode is telephone recording mode, then the call voice signal is integrated into the output result.

5. The method according to claim 1, characterized in that, Before determining the sub-band mask of the multi-channel speech signal based on the machine learning model, the method further includes: Noise suppression is performed on the first frequency domain signal using a fixed beamforming algorithm. The first frequency domain signal is obtained by frequency domain transformation of the ambient speech signal in the user's environment.

6. The method according to claim 1, characterized in that, The calculation of beam weights based on the sub-band mask using an adaptive beamforming algorithm includes: Using the multi-task learning mechanism of the machine learning model, the speech mask and noise mask of the speech signal for each channel are predicted; Based on the speech mask and the noise mask, the covariance matrix in the adaptive beamforming algorithm is constructed. The covariance matrix includes the noise covariance matrix and the speech covariance matrix corresponding to the clean speech after the speech mask. The beam weights are determined based on the covariance matrix.

7. The method according to any one of claims 1-3, 5, and 6, characterized in that, The method further includes converting the enhanced single-channel speech signal from the frequency domain to the time domain.

8. A voice enhancement device, characterized in that, include: The acquisition module is used to acquire multi-channel voice signals; A mask determination module is used to determine the sub-band mask of the multi-channel speech signal based on a machine learning model; The weight determination module is used to calculate the beam weights based on the sub-band mask using an adaptive beamforming algorithm. The speech enhancement module is used to enhance the multi-channel speech signal based on the beam weights and determine the enhanced speech signal.

9. A voice enhancement device, characterized in that, include: A microphone array and a processor, wherein the microphone array is connected to the processor. The microphone array is used to collect multi-channel voice signals; The processor is configured to execute computer execution instructions stored in memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A voice interaction system, characterized in that, include: Microphone array, used to acquire multi-channel voice signals; The processor, connected to the microphone array, is used to generate a sub-band mask for the multi-channel speech signal through a machine learning model, and to calculate beam weights based on the sub-band mask using an adaptive beamforming algorithm to enhance the multi-channel speech signal. A speech recognition unit, connected to the processor, is used to recognize and process the enhanced speech signal.

Images