Multi-channel voice wake-up method and device

Abstract

The present disclosure relates to the field of speech recognition technology, and discloses a multi-channel speech wake-up method and device. The method includes: generating multi-channel speech data based on speech data collected from multiple microphone channels; inputting the multi-channel speech data into a speech feature extraction network for layer-by-layer processing to obtain audio frame output features corresponding to the multi-channel speech data; the speech feature extraction network is a pre-trained neural network, and the speech feature extraction network includes: a beamforming module, a Mel filter, and a speech encoder connected in sequence through an output-input relationship; inputting at least part of the features of the audio frame output features corresponding to the multi-channel speech data into a beam search module for decoding to obtain a pronunciation sequence corresponding to the multi-channel speech data; when the pronunciation sequence corresponding to the multi-channel speech data contains a wake-up word, sending a speech wake-up instruction reduces the dependence on computing resources and improves the accuracy of the wake-up result.

Description

Technical Field

[0001] The present disclosure relates to the field of speech recognition technology, and in particular to a multi-channel speech wake-up method and device. Background Art

[0002] With the development of technology in the smart cockpit industry, the reliance on and requirements for voice wake-up are getting higher and higher. In order to meet more diverse user needs, automobile manufacturers often install multiple microphones in different locations inside and outside the car cabin to collect sound, hoping to enhance the experience of voice wake-up and provide richer functions such as voice noise reduction and beamforming (enhancing the voice in the direction of the wake-up person and shielding the voice in other directions).

[0003] In related technologies, voice wake-up, voice noise reduction, beamforming and other functions are developed independently, and the voice of each microphone channel is woken up separately. Therefore, related technologies need to rely on more computing resources, and the accuracy of the wake-up results needs to be improved. Summary of the invention

[0004] In view of this, the present disclosure provides a multi-channel voice wake-up method to solve the problem in the related art that more computing resources need to be relied on and the accuracy of the wake-up result needs to be improved.

[0005] In a first aspect, the present disclosure provides a multi-channel voice wake-up method, the method comprising: generating multi-channel voice data based on voice data collected from multiple microphone channels; inputting the multi-channel voice data into a voice feature extraction network for layer-by-layer processing to obtain audio frame output features corresponding to the multi-channel voice data; the voice feature extraction network is a pre-trained neural network, and the voice feature extraction network comprises: a beamforming module, a Mel filter, and a voice encoder connected in sequence through an output-input relationship; inputting at least part of the audio frame output features corresponding to the multi-channel voice data into a beam search module for decoding to obtain a pronunciation sequence corresponding to the multi-channel voice data; when a wake-up word is included in the pronunciation sequence corresponding to the multi-channel voice data, a voice wake-up instruction is sent.

[0006] In a second aspect, the present disclosure provides a multi-channel voice wake-up device, which includes: a voice data generation module, which is used to generate multi-channel voice data based on voice data collected from multiple microphone channels; an output feature generation module, which is used to input the multi-channel voice data into a voice feature extraction network for layer-by-layer processing to obtain audio frame output features corresponding to the multi-channel voice data; the voice feature extraction network is a pre-trained neural network, and the voice feature extraction network includes: a beamforming module, a Mel filter, and a voice encoder connected in sequence through an output-input relationship; a pronunciation sequence generation module, which is used to input at least part of the features of the audio frame output features corresponding to the multi-channel voice data into a beam search module for decoding to obtain a pronunciation sequence corresponding to the multi-channel voice data; a wake-up instruction sending module, which is used to send a voice wake-up instruction when a wake-up word is included in the pronunciation sequence corresponding to the multi-channel voice data.

[0007] In a third aspect, the present disclosure provides a computer device, including: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multi-channel voice wake-up method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0008] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the multi-channel voice wake-up method of the first aspect or any corresponding embodiment thereof.

[0009] In a fifth aspect, the present disclosure provides a computer program product, including computer instructions, which are used to enable a computer to execute the multi-channel voice wake-up method of the above-mentioned first aspect or any corresponding embodiment thereof.

[0010] The multi-channel voice wake-up method and device provided in the embodiments of the present disclosure use a voice feature extraction network to process multi-channel voice data layer by layer, generate audio frame output features, and use a beam search module to decode the audio frame output features to obtain a pronunciation sequence, and then send a voice wake-up instruction when the pronunciation sequence contains a wake-up word. In this process, since a jointly trained voice feature extraction network is used to process multi-channel voice data, the beamforming module generates enhanced single-channel voice data, which reduces the required computing resources, improves voice quality and voice recognition accuracy, and the Mel filter more accurately describes the characteristics of the sound signal and improves the accuracy of audio recognition. The voice encoder brings rich voice prior knowledge to the model, and then the beam search module is used to improve the accuracy of decoding and the accuracy of the generated voice wake-up instructions, thereby improving the accuracy of the wake-up results. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 is a flowchart of a multi-channel voice wake-up method according to an embodiment of the present disclosure;

[0013] Figure 2 is a schematic diagram of the training process of the speech feature extraction network of the multi-channel speech wake-up method according to an embodiment of the present disclosure;

[0014] Figure 3a is a flowchart of a training process of a speech feature extraction network in a multi-channel speech wake-up method according to an embodiment of the present disclosure;

[0015] Figure 3b Schematic diagram of the wake-up process of the speech feature extraction network in the multi-channel speech wake-up method according to the embodiment of the present disclosure

[0016] Figure 4 is a structural block diagram of a multi-channel voice wake-up device according to an embodiment of the present disclosure;

[0017] Figure 5 It is a schematic diagram of the hardware structure of the computer device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0019] According to an embodiment of the present disclosure, an embodiment of a multi-channel voice wake-up method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0020] In this embodiment, a multi-channel voice wake-up method is provided, which can be used in the above-mentioned computer system, such as a vehicle-mounted brain, a smart cockpit, etc. Figure 1is a flow chart of a multi-channel voice wake-up method according to an embodiment of the present disclosure. Figure 1 As shown, the process includes the following steps:

[0021] Step S101 : generating multi-channel voice data based on voice data collected from multiple microphone channels.

[0022] In this embodiment, the execution subject may collect voice data from multiple microphone channels located at different positions and / or different orientations (e.g., the main driver's seat, the co-driver's seat, the left rear, the right rear, the front, back, left, and right outside the vehicle), and generate multi-channel voice data based on the voice data collected from the multiple microphone channels. For example, the voice data collected from the multiple microphone channels may be directly used as the multi-channel voice data, or the voice data collected from the multiple microphone channels may be processed as needed to obtain the multi-channel voice data.

[0023] Step S102: input the multi-channel speech data into a speech feature extraction network for layer-by-layer processing to obtain audio frame output features corresponding to the multi-channel speech data.

[0024] In this embodiment, the speech feature extraction network is a pre-trained neural network, and the speech feature extraction network includes: a beamforming module, a Mel filter, and a speech encoder that are sequentially connected through an output-input relationship.

[0025] After the multi-channel speech data is input into the speech feature extraction network, the multi-channel speech data is processed in sequence by the beamforming module, the Mel filter, and the speech encoder, and the audio frame output features corresponding to the multi-channel speech data can be obtained.

[0026] Specifically, a beamforming module can be used to obtain single-channel enhanced speech based on multi-channel speech data; a Mel filter can be used to extract the features of the single-channel enhanced speech to obtain the features of the single-channel enhanced speech; and a speech encoder can be used to generate audio frame output features corresponding to the multi-channel speech data based on the features of the single-channel enhanced speech.

[0027] Step S103: input at least part of the features of the audio frame output corresponding to the multi-channel speech data into a beam search module for decoding to obtain a pronunciation sequence corresponding to the multi-channel speech data.

[0028] In this embodiment, the execution subject may use a beam search module to decode some or all of the features in the audio frame output features corresponding to the multi-channel speech data to obtain the pronunciation sequence corresponding to the multi-channel speech data.

[0029] Step S104: When the pronunciation sequence corresponding to the multi-channel voice data contains a wake-up word, a voice wake-up instruction is sent.

[0030] In this embodiment, the above-mentioned execution entity can detect whether the pronunciation sequence corresponding to the multi-channel voice data contains a wake-up word, and when the pronunciation sequence contains a wake-up word, generate a voice wake-up instruction corresponding to the included wake-up word, and send the voice wake-up instruction for wake-up.

[0031] The multi-channel voice wake-up method provided in this embodiment uses a voice feature extraction network to process multi-channel voice data layer by layer, generates audio frame output features, and decodes the audio frame output features using a beam search module to obtain a pronunciation sequence, and then sends a voice wake-up instruction when the pronunciation sequence contains a wake-up word. In this process, the beamforming module is used to generate enhanced single-channel voice data, which improves the voice quality and voice recognition accuracy. The Mel filter is used to more accurately describe the characteristics of the sound signal and improve the accuracy of audio recognition. The voice encoder is used to introduce rich voice prior knowledge into the model, and the beam search module is used to improve the decoding accuracy, thereby improving the accuracy of the voice wake-up instruction finally generated.

[0032] In some optional implementations of the present embodiment, the speech feature extraction network further includes: a following prediction module, the Mel filter and the speech encoder are connected via the following prediction module; the above step S101 includes: step S1011, based on the speech data collected from multiple microphone channels, generating multi-channel speech data including previous speech data of preset duration and main text speech data of target duration; the above step S102 includes: step S1012, processing the previous speech data and the main text speech data in sequence via a beamforming module and a Mel filter to obtain a previous enhanced speech feature and a main text enhanced speech feature; inputting the previous enhanced speech feature and the main text enhanced speech feature into the following prediction module for processing to obtain a following predicted speech feature; inputting the previous enhanced speech feature, the main text enhanced speech feature and the following predicted speech feature into the speech encoder for processing to obtain an audio frame output feature corresponding to the multi-channel speech data including a previous speech output feature, a main text speech output feature and a following speech output feature; the above step S103 includes: step S1031, inputting the main text speech output feature into a beam search module for decoding to obtain a pronunciation sequence corresponding to the multi-channel speech data.

[0033] In this implementation, the preset duration and target duration can be determined by those skilled in the art based on actual application scenarios and experience, and this disclosure does not limit this. For example, the target duration can be determined based on the length of the wake-up word speech, and the preset duration can be determined based on the length of the speech corresponding to the edge position of the output feature of the speech encoder.

[0034] Specifically, the execution subject can segment the voice data collected from multiple microphone channels according to the target duration, and obtain different text voice data after segmentation. Each text voice data can carry a pre-context voice data of a preset duration. Afterwards, the pre-context voice data and the text voice data are processed in sequence by the beamforming module and the Mel filter to obtain the pre-context enhanced voice features and the text enhanced voice features.

[0035] Afterwards, the context-enhanced speech features and the text-enhanced speech features are input into the context prediction module to obtain the context-predicted speech features generated by the context prediction module. Furthermore, the context-enhanced speech features, the text-enhanced speech features and the context-predicted speech features are input into the speech encoder to generate corresponding context-enhanced speech output features, text-enhanced speech output features and context-predicted speech output features.

[0036] Afterwards, the text speech output features can be input into the beam search module for decoding to obtain the pronunciation sequence corresponding to the text speech output features.

[0037] In the process of generating speech output features in the present implementation, it is considered that in the process of generating speech output features from multi-channel speech data, the recognition degree of features at the edge positions on both sides of the data block is lower than the recognition degree of features at the middle positions. Therefore, when processing the speech data, the main text speech data carries its previous speech data, so that it can be processed in sequence through the beamforming module and the Mel filter to generate previous context enhanced speech features and main text enhanced speech features, and based on the pre-trained following prediction module, generate following predicted speech features according to the previous context enhanced speech features and the main text enhanced speech features, and then use the speech encoder to generate corresponding previous context speech output features, main text speech output features and following speech output features according to the previous context enhanced speech features, the main text enhanced speech features and the following predicted speech features, thereby ensuring the feature recognition degree of the main text speech output features, thereby improving the accuracy of the pronunciation sequence, the accuracy of the speech wake-up instruction, and the wake-up effect of the speech wake-up.

[0038] For further information, please refer to Figure 2 , Figure 2 1 is a schematic diagram of the training process of the speech feature extraction network of the multi-channel speech wake-up method according to an embodiment of the present disclosure. The training process can be used in the above-mentioned computer system, such as a vehicle brain, a smart cockpit, etc., or other computer systems different from the computer system used in the above-mentioned multi-channel speech wake-up method, such as a computer system for model training, a cloud server, etc. Figure 2 As shown, the process includes the following steps:

[0039] Step S201, generating noisy data based on the clean speech data after adding the target noise data.

[0040] In this embodiment, the target noise data is noise data with a random signal-to-noise ratio within a preset range. Before training the speech feature extraction network, data preparation can be performed. The data preparation includes: pre-collecting a data set containing various audio signals, the data set including clean speech data and noise data collected by each microphone channel at different positions and / or different orientations (such as the main driver, co-driver, left rear, right rear, front, back, left, right, etc. in the car), wherein the clean speech data is divided into wake-up speech data and non-wake-up speech data.

[0041] Among them, clean voice data can be defined as Clean-data, which can include clean wake-up voice data Wakeup-data and clean non-wake-up voice data Reject-data; noise data can be defined as Noise-data.

[0042] After that, wake-up tags can be added to the data in the dataset. The wake-up tags can include all supported wake-up words and categories representing no wake-up. The wake-up words can be represented by the following identifier: keyword-n, where n represents the nth wake-up word. The category representing no wake-up can be represented by the identifier reject.

[0043] When training the speech feature extraction network, the noise data Noise-data can be used to add noise to the clean wake-up speech data Wakeup-data and the clean non-wake-up speech data Reject-data according to a random signal-to-noise ratio within a preset range, thereby obtaining the noisy data Speech-Augment-data for training the speech feature extraction network.

[0044] Step S202: input the noisy data into a speech feature extraction network for layer-by-layer processing to obtain audio frame output features corresponding to the noisy data.

[0045] In this embodiment, the above-mentioned execution entity can input the noisy data into the speech feature extraction network, thereby using the data processing layers connected in sequence in the speech feature extraction network to perform layer-by-layer processing to obtain the audio frame output features corresponding to the noisy data.

[0046] Specifically, when the speech feature extraction network includes a beamforming module, a Mel filter, and a speech encoder connected in sequence through an output-input relationship, the noisy data can be input into the beamforming module to obtain the single-channel enhanced sound output by the beamforming module, and the single-channel enhanced sound can be input into the Mel filter to extract the single-channel enhanced sound features corresponding to each frame of audio in the single-channel enhanced sound, and the single-channel enhanced sound features are then input into the speech encoder to obtain the audio frame output features output by the speech encoder.

[0047] The speech encoder may use a pre-trained speech recognition module to bring more abundant speech prior knowledge to the model. For example, the speech recognition module may use a technology used for speech recognition in existing technology or future technology, which is not limited in this application. In some instances, the speech recognition module may be a self-developed speech recognition tool or a third-party speech recognition tool.

[0048] Step S203: Input the audio frame output features corresponding to the noisy data into a pronunciation classification network to obtain a predicted pronunciation category of the audio frame.

[0049] In this embodiment, the above audio frame output features can be input into the pronunciation classification network to obtain the probability of the audio frame in each pronunciation category, so that the pronunciation category with the highest probability can be used as the predicted pronunciation category. The pronunciation category can include: the pronunciation category included in all wake-up words, the category sil representing silence, and the category other representing non-wake-up word pronunciation.

[0050] Step S204, updating the parameters of the pronunciation classification network and the speech feature extraction network according to the probability distribution of the audio frame in predicting the pronunciation category and the cross entropy loss of the wake-up label.

[0051] In this embodiment, the wake-up tag corresponds to the audio frame of the clean speech data corresponding to the audio frame of the noisy data. The wake-up tag is a wake-up tag added to the pre-collected data set containing various audio signals during data preparation. The wake-up tag can include all supported wake-up words and categories representing no wake-up. Among them, the wake-up word can be represented by the following identifier: keyword-n, n represents the nth wake-up word. The category representing no wake-up can be represented by the identifier reject.

[0052] The execution entity can update the parameters of the pronunciation classification network and the speech feature extraction network according to the cross entropy loss of the wake-up label quality inspection and the probability of the pronunciation category contained in the wake-up word, the category representing silence, and the category representing the non-wake-up word in the audio frame of the noisy data.

[0053] The speech feature extraction network of the multi-channel speech wake-up method provided in this embodiment, when training the speech feature extraction network, jointly trains the pronunciation classification network, and updates the parameters of the pronunciation classification network and the speech feature extraction network according to the cross entropy loss of the audio frame in predicting the probability distribution of the pronunciation category and the wake-up label, thereby simultaneously improving the performance of the pronunciation classification network and the speech feature extraction network in the process of joint training, making the classification results output by the pronunciation classification network more accurate, and the audio frame output features extracted by the speech feature extraction network more efficient in outputting accurate classification results, thereby improving the accuracy of the recognition results when the audio frame output features extracted by the speech feature extraction network are used for speech wake-up.

[0054] In some optional implementations of the above embodiments, when the speech feature extraction network includes a following prediction module, the above step S201 includes: step S2011, based on the clean speech data after adding the target noise data, generating noise-added data including preceding sound data of a preset duration, main text sound data of a target duration, and following sound data of a preset duration; the above step S202 includes: step S2021, processing the noise-added data in sequence through a beamforming module and a Mel filter to obtain enhanced sound features of an audio frame corresponding to the noise-added data, the enhanced sound features of the audio frame including: preceding enhanced sound features, main text enhanced sound features, and following enhanced sound features; the preceding sound data The enhanced sound features and the main text enhanced sound features are input into the following prediction module for processing to obtain the following predicted sound features; the mean square error loss of the following predicted sound features and the following enhanced sound features is used to update the parameters of both the following prediction module and the beamforming module; the previous text enhanced sound features, the main text enhanced sound features and the following predicted sound features are input into the speech encoder for processing to obtain the audio frame output features corresponding to the noisy data including the previous text sound output features, the main text sound output features and the following sound output features; the above step S203 includes: step S2031, inputting the main text sound output features into the pronunciation classification network to obtain the probability distribution of the audio frame in the predicted pronunciation category.

[0055] In this implementation, the clean speech data after adding the target noise data can be segmented to obtain the noisy data, which includes: the previous sound data of the preset duration, the main text sound data of the target duration, and the following sound data of the preset duration; the noisy data is processed layer by layer through the beamforming module and the Mel filter to obtain the enhanced sound features of the audio frame corresponding to the noisy data block, and the enhanced sound features include: the previous enhanced sound features, the main text enhanced sound features, and the following enhanced sound features; the previous enhanced sound features and the main text enhanced sound features are input into the following prediction module to obtain the following predicted sound features. In this process, the mean square error loss of the following predicted sound features and the following enhanced sound features can be used to update the parameters of both the following prediction module and the beamforming module.

[0056] Afterwards, using a speech encoder, the audio frame output features corresponding to the noisy data including the previous context sound output features, the main text sound output features and the following context sound output features can be generated based on the previous context enhanced sound features, the main text enhanced sound features and the following context predicted sound features.

[0057] Corresponding to the audio frame output features corresponding to the noisy data output by the speech encoder, the main text sound output features with the best feature recognition among the above sound output features, the main text sound output features and the following sound output features can be further input into the pronunciation classification network to obtain the probability distribution of the audio frame in the predicted pronunciation category.

[0058] In this implementation, by using the mean square error (MSE) loss of the following prediction sound features of the audio frame corresponding to the noisy data and the following enhanced sound features of the audio frame corresponding to the noisy data, the parameters of both the following prediction module and the beamforming module are updated, and the performance of both the following prediction module and the beamforming module can be improved through back propagation, thereby improving the accuracy of the data output by both. At the same time, in the process of training the speech feature extraction network, the following prediction module is introduced for joint training, thereby improving the accuracy of the following prediction sound features output by the following prediction module, and improving the recognizability of the text sound output features, and finally improving the accuracy of speech wake-up based on the text sound output features.

[0059] In some optional implementations of the above embodiment, the prediction module below is an improved neural network model based on the image segmentation architecture of the convolutional neural network. The improved neural network model may include: a three-level gated recurrent unit (GRU) and a one-level multilayer perceptron (MLP) connected in sequence through an output-input relationship.

[0060] In this optional implementation, the following prediction module can be an improved neural network model SimuNet based on the image segmentation architecture of the convolutional neural network.

[0061] Specifically, the first-level GRU, the second-level GRU, and the third-level GRU of the three-level gated recurrent unit included in SimuNet are connected in sequence through the output-input relationship, that is: the first-level GRU: as the input level, receives the context-enhanced speech features and the text-enhanced speech features output by the Mel filter, and performs preliminary processing and encoding on the input data to convert it into a higher-dimensional representation. The second-level GRU: further processes the output of the first-level GRU, encodes the data more deeply, and extracts more abstract and advanced features. The third-level GRU: as the last level of the GRU network, further processes and integrates the output of the second-level GRU. The output of this level will be used as the input of the MLP network.

[0062] Specifically, the multi-layer perceptron MLP includes an input layer, a hidden layer, and an output layer. The input layer receives the output from the third-level GRU, the hidden layer extracts and integrates features, and the output layer generates the final prediction result, that is, generates the following predicted speech features.

[0063] The following prediction module in this implementation combines the advantages of GRU and MLP by explicitly improving the specific structure of the neural network model SimuNet, including a three-level gated recurrent unit and a one-level multilayer perceptron, thereby ensuring that the following prediction module can process sequence data and extract high-level features. At the same time, due to its relatively simple structure and fewer parameters, the training efficiency of the model is improved.

[0064] In some optional implementations of the above embodiments, the beamforming module is implemented using a minimum variance distortion-free response algorithm based on a neural network; when training the beamforming module, the parameters of the beamforming module are updated using the mean square error loss of the single-channel enhanced speech output by the beamforming module and its corresponding clean speech data.

[0065] In this implementation, a beamforming module is used to enhance the signal strength in a specific direction and suppress interference or noise in other directions, and can be used in a microphone array to improve the quality and clarity of speech signals in a noisy environment.

[0066] The minimum variance distortionless response based on neural networks is to apply neural networks to MVDR, which can achieve adaptive learning in complex environments. Neural networks can handle nonlinear relationships and learn the best mapping from input signals (such as signals received by multiple microphone arrays) to desired outputs (such as enhanced speech signals); minimum variance distortionless response (MVDR) can minimize noise and interference in the output signal while keeping the target signal (such as speech) undistorted.

[0067] The beamforming module in this implementation method can dynamically adjust the beamforming weights based on the MVDR algorithm of the neural network to adapt to different noise conditions and target signal characteristics, thereby improving the performance and robustness of the beamforming. During training, the beamforming module uses the mean square error loss of the single-channel enhanced speech output by the beamforming module and its corresponding clean speech data to update the parameters of the beamforming module, thereby optimizing the performance of the beamforming module and improving the accuracy of the output of the beamforming module.

[0068] In some optional implementations of the above embodiment, the Mel filter uses a window of preset duration and a step of predetermined duration to calculate filter bank features of a predetermined number of channels.

[0069] In this implementation, the execution subject can calculate the filter bank Fbank features of the predetermined number of channels with a window of preset duration and a step of predetermined duration (for extracting features of one frame of audio), thereby generating Fbank features corresponding to the input data for the input data. Those skilled in the art can determine the window of preset duration, the step of predetermined duration and the predetermined number of channels according to actual application scenarios or experience.

[0070] The Mel filter in this implementation can simulate the auditory characteristics of the human ear through the design of the Mel filter group, effectively extract the speech features related to the human ear hearing, and map the original signal to a relatively small group of Mel frequency channels, thereby reducing the processing dimension of the signal and extracting the key features in the speech signal, such as resonance peaks, fundamental frequencies, etc., thereby improving the recognition performance of the system.

[0071] In a specific application scenario of the multi-channel voice wake-up method of an embodiment of the present disclosure, the specific implementation steps of the multi-channel voice wake-up method include the following:

[0072] 1. Data Preparation

[0073] 1. Collect data: Collect a data set containing various audio signals, including clean speech data and noise data collected by each microphone channel at different positions and orientations (driver's seat, co-driver's seat, left rear, right rear, front, back, left, right, etc.). Clean speech is divided into wake-up speech data and non-wake-up speech data.

[0074] Among them: clean speech data is defined as Clean-data, including clean wake-up speech data Wakeup-data and clean non-wake-up speech data Reject-data; noise data is defined as Noise-data.

[0075] 2. Label data: Add wake-up tags to the data, including all supported wake-up words keyword-n (n represents the nth wake-up word) and the category reject representing non-wake-up.

[0076] 2. Model Training

[0077] like Figure 3a As shown, Figure 3a It is a flowchart of the training process of the speech feature extraction network model in the multi-channel speech wake-up method according to an embodiment of the present disclosure.

[0078] 1. Use target noise data (noise data Noise-data with a random signal-to-noise ratio within a preset range) to add noise to the clean wake-up speech data Wakeup-data and the clean non-wake-up speech data Reject-data to obtain the noisy data Speech-Augment-data after adding the target noise data.

[0079] 2. Split the noise-added speech-augment-data into different chunks of text sound data. Each chunk is 600ms long, which is approximately equal to the length of the wake-up word speech. Each chunk carries a left-context sound data and a right-context sound data, both of which are 200ms long.

[0080] 3. Input the noisy data into the beamforming module, and the beamforming module outputs the enhanced single-channel sound data, that is, the single-channel enhanced sound (including the main text enhanced sound data chunk-aug, the upper text enhanced sound data left-context-aug, and the lower text enhanced sound data right-context-aug). The beamforming module is implemented using the minimum variance distortionless response MVDR (Minimum Variance Distortionless Response) solution based on a neural network. The minimum variance distortionless response MVDR based on a neural network is trained using the mean square error MSE loss, and the loss function is defined as Loss-beamforming. The learned true value is the clean data (Clean-data) of the channel corresponding to the sound source.

[0081] 4. Calculate the 80-channel filter group Fbank features of the single-channel enhanced sound according to a 25-ms window and a 10-ms step (used to extract one audio frame) to obtain an enhanced sound feature group, that is, for the main text enhanced sound data chunk-aug, the above enhanced sound data left-context-aug, and the following enhanced sound data right-context-aug, the corresponding Fbank features are calculated as the main text enhanced sound feature chunk-aug-feature, the above enhanced sound feature left-context-aug-feature, and the following enhanced sound feature right-context-aug-feature.

[0082] The chunk-aug-feature and left-context-aug-feature are sent to the improved context prediction module based on the U-Net architecture - the neural network model SimuNet module to predict the right-context-aug-feature. The output result is the context prediction sound feature right-context-predict-feature. The structure of SimuNet consists of a 3-level GRU network and a 1-level MLP network. MSE loss is used for training, and the loss function is defined as Loss-simu.

[0083] 5. Input left-context-aug-feature, chunk-aug-feature, and right-context-predict-feature into the speech encoder, and output the high-order feature representation of each audio frame (10ms per frame), that is, output the audio frame output features corresponding to the noise-added data. The speech encoder uses speech recognition technology pre-trained on speech recognition tasks, which can bring richer speech prior knowledge to the model. The corresponding outputs of left-context-aug-feature, chunk-aug-feature, and right-context-predict-feature are the upper sound output feature left-context-out, the main text sound output feature chunk-out, and the lower sound output feature right-context-out.

[0084] 6. Perform speech wake-up task training on chunk-out, the feature of text sound output. The method is to input chunk-out into the pronunciation classification network of the first-level MLP structure, and predict the pronunciation category of each frame, including the pronunciation category of all wake-up words, a category sil representing silence, and a category other representing non-wake-up word pronunciation. The training is performed using cross entropy loss, and the loss function is defined as Loss-KWS.

[0085] 3. Wake-up process

[0086] like Figure 3b As shown, Figure 3b It is a flowchart of the wake-up process of the speech feature extraction network in the multi-channel speech wake-up method according to an embodiment of the present disclosure.

[0087] 1. Each time, 800ms of speech data is sent to the voice wake-up system composed of a special speech feature extraction network and a beam search module. The first 200ms is left-context, which is used to provide the above information. In fact, only the wake-up judgment is made on the last 600ms chunk.

[0088] 2. left-context and chunk are input into the beamforming module and features are extracted to obtain single-channel enhanced speech chunk-aug-feature and left-context-aug-feature.

[0089] 3. Chunk-aug-feature and left-context-aug-feature are input into SimuNet for prediction to obtain right-context-predict-feature.

[0090] 4. The left-context-aug-feature, chunk-aug-feature, and right-context-predict-feature are input into the speech encoder to obtain the model output chunk-out corresponding to the chunk-aug-feature part.

[0091] 5. Perform beam search decoding on chunk-out to obtain the pronunciation sequence corresponding to the chunk. If the pronunciation sequence contains the wake-up word, wake-up is performed, otherwise, wake-up is not performed.

[0092] In this embodiment, a multi-channel voice wake-up device is also provided, which is used to implement the above embodiments and preferred implementations, and will not be repeated here. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0093] This embodiment provides a multi-channel voice wake-up device, such as Figure 4 As shown, including:

[0094] A voice data generating module 401, for generating multi-channel voice data based on voice data collected from multiple microphone channels;

[0095] The output feature generation module 402 is used to input the multi-channel speech data into the speech feature extraction network for layer-by-layer processing to obtain the audio frame output features corresponding to the multi-channel speech data; the speech feature extraction network is a pre-trained neural network, and the speech feature extraction network includes: a beamforming module, a Mel filter, and a speech encoder connected in sequence through an output-input relationship;

[0096] The pronunciation sequence generation module 403 is used to input at least part of the features of the audio frame output corresponding to the multi-channel speech data into the beam search module for decoding to obtain the pronunciation sequence corresponding to the multi-channel speech data;

[0097] The wake-up instruction sending module 404 is used to send a voice wake-up instruction when the pronunciation sequence corresponding to the multi-channel voice data contains a wake-up word.

[0098] In some optional implementations of this embodiment, the speech feature extraction network in the output feature generation module further includes: a following prediction module, the Mel filter and the speech encoder are connected via the following prediction module;

[0099] The voice data generating module 401 is further used to: generate multi-channel voice data including context voice data of preset duration and body voice data of target duration based on the voice data collected from multiple microphone channels;

[0100] The output feature generation module 402 is further used to: process the preceding speech data and the main text speech data in sequence through a beamforming module and a Mel filter to obtain preceding speech enhancement features and main text enhancement features; input the preceding speech enhancement features and main text enhancement features into a following prediction module for processing to obtain following predicted speech features; input the preceding speech enhancement features, main text enhancement features and following predicted speech features into a speech encoder for processing to obtain audio frame output features corresponding to multi-channel speech data including preceding speech output features, main text speech output features and following speech output features;

[0101] The pronunciation sequence generation module 403 is further used to: input the text speech output features into the beam search module for decoding, and obtain the pronunciation sequence corresponding to the multi-channel speech data.

[0102] In some optional implementations of this embodiment, the speech feature extraction network in the output feature generation module 402 is trained using the following modules (not shown in the figure):

[0103] A noise-added data generation module, used to generate noise-added data based on clean speech data after adding target noise data, where the target noise data is noise data with a random signal-to-noise ratio within a preset range;

[0104] A noise-added data processing module is used to input the noise-added data into a speech feature extraction network for layer-by-layer processing to obtain audio frame output features corresponding to the noise-added data;

[0105] A pronunciation category prediction module, used to input at least part of the features of the audio frame output corresponding to the noisy data into a pronunciation classification network to obtain a predicted pronunciation category of the audio frame;

[0106] A network parameter update module is used to update the parameters of the pronunciation classification network and the speech feature extraction network according to the cross entropy loss of the predicted pronunciation category and the wake-up label; the wake-up label corresponds to the audio frame of the clean speech data corresponding to the audio frame of the noisy data.

[0107] In some optional implementations of the present embodiment, when the noise-added data processing module includes a following prediction module in the speech feature extraction network, the following prediction module is an improved neural network model based on the image segmentation architecture of a convolutional neural network, and the following prediction module includes: a three-level gated recurrent unit and a one-level multilayer perceptron connected in sequence through an output-input relationship.

[0108] In some optional implementations of this embodiment, the noise-added data generation module is further used to: generate noise-added data including preceding sound data of a preset duration, main sound data of a target duration, and following sound data of a preset duration based on the clean speech data after adding the target noise data;

[0109] The noisy data processing module is further used to: input the noisy data into the speech feature extraction network for layer-by-layer processing to obtain the audio frame output features corresponding to the noisy data, including: processing the noisy data in sequence through the beamforming module and the Mel filter to obtain the enhanced sound features of the audio frame corresponding to the noisy data, the enhanced sound features of the audio frame including: the enhanced sound features of the previous text, the enhanced sound features of the main text and the enhanced sound features of the following text; input the enhanced sound features of the previous text and the enhanced sound features of the main text into the following text prediction module for processing to obtain the predicted sound features of the following text; use the mean square error loss of the predicted sound features of the following text and the enhanced sound features of the following text to update the parameters of the following text prediction module and the beamforming module; input the enhanced sound features of the previous text, the enhanced sound features of the main text and the predicted sound features of the following text into the speech encoder for processing to obtain the audio frame output features corresponding to the noisy data including the sound output features of the previous text, the sound output features of the main text and the sound output features of the following text;

[0110] The pronunciation category prediction module is further used to: input at least part of the audio frame output features corresponding to the noisy data into the pronunciation classification network to obtain the predicted pronunciation category of the audio frame, including: inputting the text sound output features into the pronunciation classification network to obtain the predicted pronunciation category of the audio frame.

[0111] In some optional implementations of the present embodiment, the beamforming module in the speech feature extraction network of the noisy data processing module is implemented using a minimum variance distortion-free response algorithm based on a neural network; when training the beamforming module, the parameters of the beamforming module are updated using the mean square error loss of the single-channel enhanced speech output by the beamforming module and its corresponding clean speech data.

[0112] In some optional implementations of this embodiment, the Mel filter in the speech feature extraction network of the noise-added data processing module uses a window of preset duration and a step of predetermined duration to calculate filter group features of a predetermined number of channels.

[0113] The further functional description of each of the above modules is the same as that of the above corresponding embodiments and will not be repeated here.

[0114] The multi-channel voice wake-up device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0115] See also Figure 5 , Figure 5 1 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present disclosure, and the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).

[0116] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0117] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0118] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0119] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0120] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0121] A part of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0122] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A multi-channel voice wake-up method, characterized in that: The method comprises: generating multi-channel voice data based on voice data collected from a plurality of microphone channels; Inputting the multi-channel speech data into a speech feature extraction network for layer-by-layer processing to obtain audio frame output features corresponding to the multi-channel speech data; the speech feature extraction network is a pre-trained neural network, and the speech feature extraction network includes: a beamforming module, a Mel filter, and a speech encoder connected in sequence through an output-input relationship; Inputting at least part of the features of the audio frame output corresponding to the multi-channel speech data into a beam search module for decoding to obtain a pronunciation sequence corresponding to the multi-channel speech data; When a wake-up word is included in the pronunciation sequence corresponding to the multi-channel voice data, a voice wake-up instruction is sent.

2. The method according to claim 1, characterized in that The speech feature extraction network further includes: a following prediction module, the Mel filter and the speech encoder are connected via the following prediction module; The generating multi-channel voice data based on the voice data collected from the multiple microphone channels includes: generating multi-channel voice data including context voice data of a preset duration and body voice data of a target duration based on the voice data collected from the multiple microphone channels; The step of inputting the multi-channel speech data into a speech feature extraction network for layer-by-layer processing to obtain audio frame output features corresponding to the multi-channel speech data includes: sequentially processing the above speech data and the main text speech data via the beamforming module and the Mel filter to obtain the above enhanced speech features and the main text enhanced speech features; inputting the above enhanced speech features and the main text enhanced speech features into the following prediction module for processing to obtain the following predicted speech features; inputting the above enhanced speech features, the main text enhanced speech features and the following predicted speech features into the speech encoder for processing to obtain the audio frame output features corresponding to the multi-channel speech data including the above speech output features, the main text speech output features and the following speech output features; The step of inputting at least part of the audio frame output features corresponding to the multi-channel speech data into a beam search module for decoding to obtain a pronunciation sequence corresponding to the multi-channel speech data comprises: inputting the text speech output features into a beam search module for decoding to obtain a pronunciation sequence corresponding to the multi-channel speech data.

3. The method according to claim 1 or 2, characterized in that: The speech feature extraction network is trained by the following steps: Generating noise-added data based on the clean speech data after adding target noise data, wherein the target noise data is noise data with a random signal-to-noise ratio within a preset range; Inputting the noise-added data into the speech feature extraction network for layer-by-layer processing to obtain audio frame output features corresponding to the noise-added data; Inputting at least some of the features of the audio frame output corresponding to the noisy data into a pronunciation classification network to obtain a predicted pronunciation category of the audio frame; The parameters of the pronunciation classification network and the speech feature extraction network are updated according to the cross entropy loss of the predicted pronunciation category and the wake-up label; the wake-up label corresponds to the audio frame of the clean speech data corresponding to the audio frame of the noisy data.

4. The method according to claim 3, characterized in that: When the speech feature extraction network includes a following prediction module, the generating of the noise-added data based on the clean speech data after adding the target noise data includes: generating the noise-added data including the preceding sound data of a preset duration, the main text sound data of a target duration, and the following sound data of a preset duration based on the clean speech data after adding the target noise data; The step of inputting the noisy data into the speech feature extraction network for layer-by-layer processing to obtain the audio frame output features corresponding to the noisy data comprises: processing the noisy data in sequence via the beamforming module and the Mel filter to obtain the enhanced sound features of the audio frame corresponding to the noisy data, wherein the enhanced sound features of the audio frame include: the enhanced sound features of the preceding text, the enhanced sound features of the main text, and the enhanced sound features of the following text; inputting the enhanced sound features of the preceding text and the enhanced sound features of the main text into the following text prediction module for processing to obtain the predicted sound features of the following text; using the mean square error loss of the predicted sound features of the following text and the enhanced sound features of the following text to update the parameters of both the following text prediction module and the beamforming module; inputting the enhanced sound features of the preceding text, the enhanced sound features of the main text, and the predicted sound features of the following text into the speech encoder for processing to obtain the audio frame output features corresponding to the noisy data including the sound output features of the preceding text, the sound output features of the main text, and the sound output features of the following text; The step of inputting at least part of the audio frame output features corresponding to the noisy data into a pronunciation classification network to obtain a predicted pronunciation category of the audio frame includes: inputting the text sound output features into a pronunciation classification network to obtain a predicted pronunciation category of the audio frame.

5. The method according to claim 4, characterized in that The following prediction module is an improved neural network model based on the image segmentation architecture of the convolutional neural network. The following prediction module includes: a three-level gated recurrent unit and a one-level multilayer perceptron that are sequentially connected through an output-input relationship.

6. The method according to claim 3, characterized in that The beamforming module is implemented by using a minimum variance distortion-free response algorithm based on a neural network; When the beamforming module is trained, the mean square error loss between the single-channel enhanced speech output by the beamforming module and the corresponding clean speech data is used to update the parameters of the beamforming module.

7. The method according to claim 3, characterized in that The Mel filter uses a window of preset duration and a step of predetermined duration to calculate filter bank features of a predetermined number of channels.

8. A multi-channel voice wake-up device, characterized in that: The device comprises: A voice data generation module, for generating multi-channel voice data based on voice data collected from multiple microphone channels; An output feature generation module is used to input the multi-channel speech data into a speech feature extraction network for layer-by-layer processing to obtain audio frame output features corresponding to the multi-channel speech data; the speech feature extraction network is a pre-trained neural network, and the speech feature extraction network includes: a beamforming module, a Mel filter, and a speech encoder connected in sequence through an output-input relationship; A pronunciation sequence generation module, used for inputting at least part of the features of the audio frame output corresponding to the multi-channel voice data into a beam search module for decoding, so as to obtain a pronunciation sequence corresponding to the multi-channel voice data; The wake-up instruction sending module is used to send a voice wake-up instruction when the pronunciation sequence corresponding to the multi-channel voice data contains a wake-up word.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multi-channel voice wake-up method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the multi-channel voice wake-up method according to any one of claims 1 to 7.

Images