A speech recognition method and device for microphone array conference equipment

Abstract

The present application relates to the field of intelligent voice interaction technology, and discloses a voice recognition method and device for microphone array conference equipment, the method comprising obtaining input audio of any single channel; using a preset 3A model to perform frequency band modeling on the input audio, fusing the extracted features, and performing complex network calculations to obtain a mask to act on the input audio to obtain the target audio; inputting the target audio into a preset voice wake-up model to determine the audio segment of the wake-up word; judging whether there is a human voice; if there is a human voice, using a preset extraction model to extract the wake-up person's voiceprint features and determine the audio segment of the wake-up person; inputting the audio segment of the wake-up person of the current frame into a preset human voice recognition model for human voice command recognition, obtaining the human voice command of the wake-up person, identifying the intention corresponding to the human voice command through a language model, and responding to the human voice command of the wake-up person. The present application has the effect of improving the recognition accuracy of human voice commands and wake-up words in scenarios where multiple people are speaking in a conference room.

Description

Technical Field

[0001] The present application relates to the field of intelligent voice interaction technology, and in particular to a voice recognition method and device for microphone array conferencing equipment. Background Art

[0002] At present, the conference equipment array collects audio, which is then sent to the voice wake-up model and command word recognition model after simple processing to identify the corresponding voice commands and realize relatively simple voice assistant tasks.

[0003] In the existing technology, the conference equipment in the conference room has a low signal-to-noise ratio in the scenario where multiple people are talking in the conference room because the collected sound contains large echoes, reverberations and noise. It is impossible to pick up the sound clearly and to separate the specified voice commands, which causes great interference in the recognition of voice commands and has a great impact on the recognition accuracy of voice commands and wake-up words.

[0004] Regarding the above-mentioned related technologies, the inventors found that the existing sound pickup effect is poor in the scenario where multiple people are speaking in a conference room, and the specified voice commands cannot be separated, and the recognition accuracy of voice commands and wake-up words is low. Summary of the Invention

[0005] In order to improve the recognition accuracy of human voice commands and wake-up words, the present application provides a speech recognition method and device for a microphone array conference device.

[0006] In a first aspect, the present application provides a speech recognition method for a microphone array conferencing device.

[0007] This application is achieved through the following technical solutions:

[0008] A speech recognition method for a microphone array conference device comprises the following steps:

[0009] Get any single channel of speech data to be recognized and get the input audio;

[0010] Using a preset 3A model to perform noise reduction on the input audio to obtain a target audio;

[0011] The 3A model performs frequency band modeling on the input audio, fuses the extracted features, and calculates them through a complex network to obtain a mask, which is applied to the input audio to obtain the target audio;

[0012] Input the target audio into a preset voice wake-up model, output whether a wake-up word exists and the wake-up rate of the wake-up word, and determine the audio segment of the wake-up word;

[0013] Determine whether there is a human voice based on the audio segment of the wake-up word;

[0014] If there is a human voice, a preset extraction model is used to extract the wake-up voiceprint features in the audio segment of the wake-up word in which the human voice is present, to determine whether the current frame is the voice of the wake-up person, and to determine the audio segment of the wake-up person;

[0015] The audio segment of the current frame for waking up the person is input into a preset voice recognition model for voice command recognition to obtain the voice command for waking up the person, the intention corresponding to the voice command is identified through the language model, and the voice command for waking up the person is responded to.

[0016] In a preferred example, the present application can be further configured as follows:

[0017] The 3A model includes two Conv2d layers, one LSTM layer and one concat layer;

[0018] An input end of one of the Conv2d layers receives the signal after linear echo cancellation, and an output end is connected to the input end of the concat layer;

[0019] An input end of another Conv2d layer receives a linearly predicted echo signal and is connected to an input end of the concat layer via the LSTM layer;

[0020] The 3A model also includes three groups of transfer modules consisting of an LSTM layer, an LN layer and a Bi-LSTM layer connected in series in sequence. The three groups of transfer modules are arranged in series, and the output end of the concat layer is connected to the input end of the first group of transfer modules.

[0021] In a preferred example, the present application can be further configured as follows: when the 3A model is trained, the following steps are included:

[0022] A complex spectrum compression loss function is used for model training. By designing the compression rate of the complex spectrum compression loss function and the loss ratios of echo suppression, noise suppression and near-end sound quality, echo suppression, noise reduction and dereverberation of the target audio are performed.

[0023] In a preferred example, the present application can be further configured as follows: the voice wake-up model includes:

[0024] 3 translation modules consisting of a Transformer layer connected in series with an LSTM layer, a CTC fully connected layer, and a softmax layer;

[0025] The three groups of translation modules are arranged in series, and the output end of the last group of translation modules is connected to the input end of the CTC fully connected layer, and the output end of the CTC fully connected layer is connected to the input end of the softmax layer.

[0026] In a preferred example, the present application can be further configured as follows: the extraction model includes:

[0027] Two groups of extraction modules are composed of a Conv2d layer, a Bi-LSTM layer, and a Pool layer connected in series. The input end of the first group of extraction modules receives the FBANK features in the audio segment of the wake-up word, and the output end of the second group of extraction modules outputs the voiceprint features of the wake-up person.

[0028] In a preferred example, the present application can be further configured as follows: the human voice recognition model includes a streaming inference model and a non-streaming inference model, and the recognition process of the human voice recognition model includes:

[0029] Initialize the model by designing the corresponding keywords in the preset entity slot information list into hot words and loading them into the non-streaming reasoning model;

[0030] Inputting the audio segment of the current frame for waking up the person into the streaming inference model for real-time speech recognition, and outputting real-time text data;

[0031] If the audio segment for waking up the person in the current frame is the last frame, all the previous audio segments for waking up the person are integrated, input into the non-streaming reasoning model for recognition, reasoning and search, and the complete text data is output.

[0032] In a preferred example, the present application can be further configured as follows:

[0033] According to a preconfigured intent list, a finite state machine is constructed based on the slots and entity information in the intent list to query whether corresponding entity slots exist in the real-time text data and the complete text data;

[0034] At the same time, if there is a corresponding physical slot, a search tree is constructed according to the physical slot;

[0035] The physical slot is used to perform a depth-first search in the search tree to obtain the corresponding intention and the required slot information and return it, while responding to the voice command of the wake-up person.

[0036] In a preferred example, the present application may be further configured as follows: before the step of querying whether there is a corresponding physical slot in the real-time text data and the complete text data, the step further includes:

[0037] The Chinese numeral expressions representing quantities in the real-time text data and the complete text data are converted into Arabic numerals, and modal particles in the real-time text data and the complete text data are filtered.

[0038] In a preferred example, the present application may be further configured as follows: after the step of determining whether there is a human voice according to the audio segment of the wake-up word, the step further includes:

[0039] Mapping the audio segment of the wake-up word containing human voice to multi-channel audio data;

[0040] Determine the direction of the wake-up voice on the multi-channel audio data using an offline DOA algorithm to obtain the azimuth of the wake-up voice;

[0041] Based on the azimuth of the wake-up voice, a beamforming weight vector is constructed, a minimum variance distortionless response beamformer is applied to perform beamforming, and array output power variance is obtained;

[0042] Minimizing the power variance so that the response of the beamformer in the azimuth direction of the wake-up voice is maximized and the response in other directions is minimized, thereby obtaining preprocessed audio;

[0043] The mask of the 3A model is reused on the preprocessed audio to obtain and update the audio segment of the wake-up word containing human voice.

[0044] In a second aspect, the present application provides a speech recognition device for a microphone array conferencing device.

[0045] This application is achieved through the following technical solutions:

[0046] A speech recognition device for microphone array conference equipment, comprising:

[0047] The acquisition module is used to obtain the speech data to be recognized in any single channel to obtain the input audio;

[0048] a noise reduction module, configured to perform noise reduction processing on the input audio using a preset 3A model to obtain target audio; the 3A model performs frequency band modeling on the input audio, fuses the extracted features, and calculates a mask through a complex network; the mask is applied to the input audio to obtain the target audio;

[0049] A wake-up word detection module is used to input the target audio into a preset voice wake-up model, output whether a wake-up word exists and the wake-up rate of the wake-up word, and determine the audio segment of the wake-up word;

[0050] A human voice detection module is used to determine whether there is a human voice based on the audio segment of the wake-up word;

[0051] A voiceprint matching module is used to extract the wake-up person's voiceprint features in the audio segment of the wake-up word containing the human voice using a preset extraction model when a human voice is present, determine whether the current frame is the wake-up person's voice, and determine the audio segment of the wake-up person;

[0052] The human voice command recognition module is used to input the audio segment of the current frame for waking up the person into a preset human voice recognition model for human voice command recognition, obtain the human voice command for waking up the person, identify the intention corresponding to the human voice command through the language model, and respond to the human voice command for waking up the person.

[0053] In summary, compared with the prior art, the technical solution provided by this application has at least the following beneficial effects:

[0054] Acquire any single-channel voice data to be recognized and obtain the input audio, and first perform front-end processing on a single-channel audio data of the multi-channel to reduce the processing difficulty; use the preset 3A model to perform frequency band modeling on the input audio, fuse the extracted features, and calculate through the complex network to obtain a mask to act on the input audio, which can reduce the echo, reverberation and noise in the audio, obtain cleaner audio data, and achieve clear sound pickup, which is conducive to improving the recognition accuracy of human voice commands and wake-up words in subsequent audio; input the target audio into the preset voice wake-up model to determine whether there is a wake-up word in the current voice segment, output whether there is a wake-up word and the wake-up rate of the wake-up word, and determine the audio segment of the wake-up word to reduce the situation where the wake-up word is missed due to no matching human voice; then according to the audio of the wake-up word frequency band to determine whether there is a human voice; if there is a human voice, the preset extraction model is used to extract the wake-up person's voiceprint features in the audio segment of the wake-up word where there is a human voice, and determine whether the current frame is the voice of the wake-up person. By comparing the voiceprint features, the voice of the wake-up person is identified, thereby separating the specified human voice command, and determining the audio segment of the wake-up person, so as to reduce the interference to the recognition of human voice commands in the scenario where multiple people are speaking in the conference room; finally, the audio segment of the wake-up person in the current frame is input into the preset human voice recognition model for human voice command recognition, and the human voice command of the wake-up person is obtained, and then the intention corresponding to the human voice command is identified through the language model, and the human voice command of the wake-up person is responded to; it can separate human voice commands in the low signal-to-noise ratio scenario where multiple people are speaking in the conference room, and improve the recognition accuracy of human voice commands and wake-up words. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of the main flow of a speech recognition method for a microphone array conference device provided as an exemplary embodiment of the present application.

[0056] Figure 2 A structural block diagram of a 3A model of a speech recognition method for a microphone array conferencing device provided as another exemplary embodiment of the present application.

[0057] Figure 3 A structural block diagram of a voice wake-up model of a voice recognition method for a microphone array conferencing device provided in another exemplary embodiment of the present application.

[0058] Figure 4A structural block diagram of an extraction model of a speech recognition method for a microphone array conferencing device provided as an exemplary embodiment of the present application.

[0059] Figure 5 A schematic diagram of a grouped LSTM for a speech recognition method for a microphone array conferencing device provided as an exemplary embodiment of the present application.

[0060] Figure 6 A structural block diagram of a speech recognition device for a microphone array conference device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0061] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed, but as long as they are within the scope of the claims of the present application, they are protected by the patent law.

[0062] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0063] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document, unless otherwise specified, generally indicates an "or" relationship between the related objects.

[0064] The embodiments of the present application are described in further detail below with reference to the accompanying drawings.

[0065] Reference Figure 1 , an embodiment of the present application provides a speech recognition method for a microphone array conference device, and the main steps of the method are described as follows.

[0066] S1: Obtain any single channel of speech data to be recognized and obtain input audio;

[0067] S2: Using a preset 3A model to perform noise reduction on the input audio to obtain a target audio;

[0068] S3: The 3A model performs frequency band modeling on the input audio, fuses the extracted features, and calculates them through a complex network to obtain a mask, which is applied to the input audio to obtain the target audio;

[0069] S4: Input the target audio into a preset voice wake-up model, output whether there is a wake-up word and the wake-up rate of the wake-up word, and determine the audio segment of the wake-up word;

[0070] S5: Determine whether there is a human voice based on the audio segment of the wake-up word;

[0071] S6: If there is a human voice, use a preset extraction model to extract the wake-up person voiceprint features in the audio segment of the wake-up word containing the human voice, determine whether the current frame is the wake-up person's voice, and determine the audio segment of the wake-up person;

[0072] S7: Input the audio segment of the current frame for waking up the person into the preset human voice recognition model for human voice command recognition, obtain the human voice command for waking up the person, identify the intention corresponding to the human voice command through the language model, and respond to the human voice command for waking up the person.

[0073] Specifically, first collect the input voice data of any single-channel microphone mic in the multi-channel to obtain the input audio,

[0074] It is used to perform 3A front-end processing on a single channel of multi-channel data, and even perform noise reduction processing on the multi-channel data, which is conducive to better identification of human voice commands in audio data.

[0075] Reference Figure 2 Before inputting the input audio into the 3A model, the collected single-channel microphone input voice signal is first Fourier transformed, and the preset reference signal is also Fourier transformed. Then, the transformed results of the input voice signal and the reference signal are subjected to linear echo cancellation.

[0076] When performing linear echo cancellation, a frequency-domain adaptive filter can be used. Based on the transformation results of the input speech signal and the reference signal, the minimum mean square error or normalization is used to continuously adjust the filter coefficients to minimize the echo. The speech signal after linear echo cancellation and the linearly predicted echo signal are obtained, and the speech signal after linear echo cancellation is used as the input audio.

[0077] The input audio is denoised using a preset 3A model, and frequency band modeling is performed on the input audio and reference signal. The extracted features are fused using full-band and sub-band modeling. A mask is obtained through complex network calculation, and the mask is applied to the input audio to obtain the target audio.

[0078] Reference Figure 2Specifically, the 3A model includes two Conv2d layers, one LSTM layer, and one concat layer, wherein the input end of one Conv2d layer receives the signal after linear echo cancellation, and the output end is connected to the input end of the concat layer; the input end of the other Conv2d layer receives the linearly predicted echo signal, and is connected to the input end of the concat layer through the LSTM layer;

[0079] The 3A model also includes three groups of transfer modules consisting of LSTM layers, LN layers and Bi-LSTM layers connected in series. The three groups of transfer modules are arranged in series, the output end of the concat layer is connected to the input end of the first group of transfer modules, and the last group of transfer modules outputs a mask.

[0080] Apply the mask to the input audio and output a clean audio signal to obtain the target audio.

[0081] In one embodiment, in order to perform echo cancellation, noise reduction and dereverberation processing on the input audio, the 3A model training includes the following steps:

[0082] A complex spectrum compression loss function is used for model training. By designing the compression rate of the complex spectrum compression loss function and the loss ratios of echo suppression, noise suppression and near-end sound quality, echo suppression, noise reduction and dereverberation of the target audio are performed.

[0083] During training, different loss functions are used for echo and near-end sound quality to control the strength of fitting to the target audio; the echo is restricted to the greatest extent possible to reduce its interference with voice command recognition; because noise will be added during subsequent voice recognition model training, the noise suppression intensity does not need to be particularly strict during front-end processing; it is sufficient to minimize noise without affecting near-end sound quality; reverberation is strongly suppressed because it does not affect speech recognition; this is to meet the output audio requirements of the voice recognition model.

[0084] Take the complex spectrum compression loss function for echo suppression as an example:

[0085] L=0.125·L mag +0.875·L real +0.875·L imag

[0086]

[0087] Where, L mag Indicates the amplitude loss of the echo, L real Represents the real part loss of the echo, L imagrepresents the imaginary part loss of the echo, and MSE represents the mean square error between the echo signal A and the reference signal B, which is defined as:

[0088]

[0089] Wherein, n represents the number of echo signals.

[0090] In this embodiment, the complex spectrum compression loss function for suppressing noise can be designed as:

[0091]

[0092]

[0093] Where, represents the amplitude loss of noise, represents the real part loss of noise, Represents the imaginary part loss of noise.

[0094] In this embodiment, the loss of near-end sound quality is related to the dereverberation process, so the complex spectrum compression loss function for suppressing reverberation can be designed as:

[0095]

[0096] Where, Indicates the amplitude loss of the near-end audio. represents the real part loss of near-end audio, Indicates the imaginary part loss of the near-end audio.

[0097] The 3A model performs echo cancellation, noise reduction, and dereverberation processing on the input audio based on streaming inference. This makes audio processing more user-friendly, with no latency and faster computing performance. It can achieve ideal sound quality with less echo, noise, and reverberation, which is conducive to subsequent accurate speech recognition.

[0098] Next, the target audio is input into the preset voice wake-up model to determine whether there is a wake-up word in the current voice segment, output whether there is a wake-up word and the wake-up rate of the wake-up word, and determine the audio segment of the wake-up word.

[0099] Reference Figure 3 In one embodiment, the voice wake-up model includes:

[0100] 3 translation modules consisting of a Transformer layer connected in series with an LSTM layer, a CTC fully connected layer, and a softmax layer;

[0101] The three groups of translation modules are arranged in series, and the output end of the last group of translation modules is connected to the input end of the CTC fully connected layer, and the output end of the CTC fully connected layer is connected to the input end of the softmax layer.

[0102] The voice wake-up model recognizes wake-up words based on phonetic probability. It includes an encoder, CTC (Connectionist Temporal Classification), and softmax. This allows for more wake-up words, improving the fixed nature of existing wake-up words and task instructions, which often result in poor applicability. Furthermore, there's no need to retrain the model when changing wake-up words. The encoder encodes the input target audio signal, representing the translation module in the figure. CTC addresses the issue of varying input and output sequence lengths and misalignment. Finally, the softmax layer normalizes the input signal and outputs the probability of pinyin recognition for the wake-up word in the current frame to determine the audio segment of the wake-up word.

[0103] Furthermore, the voice wake-up model can also add a wake-up word setting module. Through embedding technology, the wake-up word passes through a mapping layer designed by a convolutional layer to obtain a feature vector with the same scale as the target audio after encoding. Feature fusion is performed and the final wake-up word probability is obtained through a CTC layer. This can greatly improve the wake-up rate of the voice wake-up model without increasing the false alarm rate, and the recognition accuracy of the voice wake-up model is higher.

[0104] Furthermore, before the target audio is input into the voice wake-up model, the nonlinear response curve of the human ear to the sound spectrum is simulated to extract the FBANK features of the target audio, so that the extracted features are closer to the characteristics of the human ear.

[0105] Furthermore, the start and end timestamps of the wake-up words are counted and position-coded for later use.

[0106] After determining the audio segment for the wake-up word, first determine whether a human voice is present. This determination can be made by converting the audio signal to the frequency domain for analysis, observing its spectral characteristics to determine the presence of a human voice. Human voices are generally concentrated in the frequency range of 300Hz to 3400Hz. Alternatively, an existing VAD (Voice Activity Detection) model can be used to directly identify whether a specific human voice is present in the audio signal segment, thereby determining the probability that the current speech frame is a human voice.

[0107] If there is a human voice, the preset extraction model is used to extract the wake-up voiceprint features in the audio segment of the wake-up word with a human voice, determine whether the current frame is the sound of the wake-up person, and determine the audio segment of the wake-up person.

[0108] Reference Figure 4 In one embodiment, the extraction model includes:

[0109] Two groups of extraction modules are composed of a Conv2d layer, a Bi-LSTM layer, and a Pool layer connected in series. The input end of the first group of extraction modules receives the FBANK features in the audio segment of the wake-up word, and the output end of the second group of extraction modules outputs the voiceprint features of the wake-up person.

[0110] The extraction model is designed through a convolutional network. The FBANK features in the audio segment of the wake-up word are input into the extraction model. By calculating the correlation between the preset voiceprint features of the wake-up person and the voiceprint features of the current speech frame, it is confirmed whether the current frame is the voice of the wake-up person. The output vector is the voiceprint feature of the wake-up person.

[0111] By comparing the streaming VAD and streaming voiceprint features, the voice of the wake-up person is identified, thereby separating the specified human voice command and determining the audio segment that wakes the person up.

[0112] Finally, the current frame's audio segment of the wake-up call is fed into a pre-set voice recognition model for voice command recognition. This generates the wake-up call, which is then identified using a language model to identify the intent behind the call and respond to the wake-up call. This streaming audio data is fed into the voice recognition model for speech recognition. The long processing window of speech recognition allows for better extraction of voiceprint information, thereby determining whether the target speaker is present in the current time window.

[0113] In one embodiment, the human voice recognition model includes a streaming inference model and a non-streaming inference model. The recognition process of the human voice recognition model includes:

[0114] Initialize the model by designing the corresponding keywords in the preset entity slot information list into hot words and loading them into the non-streaming reasoning model;

[0115] Inputting the audio segment of the current frame for waking up the person into the streaming inference model for real-time speech recognition, and outputting real-time text data;

[0116] If the audio segment for waking up the person in the current frame is the last frame, all the previous audio segments for waking up the person are integrated, input into the non-streaming reasoning model for recognition, reasoning and search, and the complete text data is output.

[0117] Specifically, the voice recognition model includes a streaming inference model and a non-streaming inference model. During model initialization, the corresponding keywords in the entity slot information list are designed as hot words and loaded into the non-streaming inference model. The audio cut by VAD is streamed into the voice recognition model, and the streaming inference model performs real-time speech recognition output. When the VAD cut is determined to be the last frame, the entire previous speech segment is subjected to non-streaming speech recognition inference and beamsearch based on hot word bias. Non-streaming speech recognition is then re-performed on the entire speech segment, which provides richer contextual information and more accurate recognition rates. At the same time, in order to more accurately hit relevant voice commands, a hot word bias is added, which increases the probability of command word recall in the speech recognition model inference, thereby obtaining more accurate and voice-command-friendly text and sending it to the next stage for command recognition.

[0118] The human voice recognition model uses a combination of non-streaming and streaming recognition to focus the back-end recognized audio on the instructions for waking people up, while filtering out interfering sounds from other people.

[0119] Reference Figure 5 The human voice recognition model incorporates grouped LSTM and grouped fully connected layers. By dividing the model into N groups, where N is a positive integer, the number of multiplication and addition operations can be reduced by a factor of N, thus improving computing performance. Furthermore, the number of matrix multiplication units after grouping is adapted to the number and width of CPU registers, resulting in faster computing performance and better application results in small models.

[0120] In one embodiment, the following steps are also included:

[0121] According to a preconfigured intent list, a finite state machine is constructed based on the slots and entity information in the intent list to query whether corresponding entity slots exist in the real-time text data and the complete text data;

[0122] At the same time, if there is a corresponding physical slot, a search tree is constructed according to the physical slot;

[0123] The physical slot is used to perform a depth-first search in the search tree to obtain the corresponding intention and the required slot information and return it, while responding to the voice command of the wake-up person.

[0124] In one embodiment, the step of constructing a search tree according to the physical slots includes:

[0125] The physical slots are sorted in sequence according to optional slots and mandatory slots using the slot sorting information in the intent list, and the intent list includes different intents corresponding to the same slot information.

[0126] When building the search tree, each intent template has a corresponding slot order, which must be followed in the specified order: optional slots and required slots. Furthermore, different intents may share the same slots. The input command is searched, and the order of the optional and required slots is checked, along with the hits.

[0127] In the language model, based on a pre-configured intent list, a finite state machine is constructed using the slots and entity information in the intent list, enabling rapid querying of the existence of corresponding entities in the text. Simultaneously, a search tree is constructed based on the required entity slots in the intent list. Based on the identified entities, a depth-first search is performed in the search tree to identify the corresponding intent. The corresponding intent and the required slot information are then returned to the logic execution layer.

[0128] In one embodiment, before the step of querying whether there are corresponding physical slots in the real-time text data and the complete text data, the method further includes:

[0129] The Chinese numeral expressions representing quantities in the real-time text data and the complete text data are converted into Arabic numerals, and modal particles in the real-time text data and the complete text data are filtered.

[0130] The voice recognition model is used to identify pre-set voice commands. It converts voice commands into text data. It then runs the input text data through ITN technology, converting Chinese numerical expressions representing quantities into Arabic numerals while filtering out modal particles.

[0131] In one embodiment, in order to recognize human voice commands that have the same intention but do not exist in the system, a language model with higher generalization ability is designed to perform secondary recognition on unrecognized commands, including:

[0132] Fine-tune the language model using more training data, including the following steps:

[0133] Input a random list of intents, generalized human voice commands, and multi-turn conversation data;

[0134] During inference, obtain the text that needs to be parsed, the list of intents supported by the execution layer, and the required variable slot information;

[0135] Output the corresponding intent and slot information.

[0136] In this embodiment, the language model may adopt a hidden Markov model, and the generalized human voice command-corresponding intent-corresponding slot information includes:

[0137] Display command: "Help me join the meeting at 812345", the recognized intent is "joinmeeting", and the slot is "meetId=8123456";

[0138] Implicit instruction: "I feel a bit cold", the recognized intent is "turnOn airCond", and the slot is "temp=norm=1";

[0139] Multi-round command: User: "Help me join the meeting at 812345", the recognized intent is "joinmeeting", and the slot is "meetId=8123456";

[0140] By designing more generalized voice command templates, we can generate richer training data to improve the robustness and contextual understanding capabilities of the language model.

[0141] An example of multi-round dialogue data design is as follows:

[0142] The system responds: "Please enter your password." The user responds: "No, it's 812357." The recognized intent is "joinmeeting" and the slot is "meetId=8123457," not a password intent.

[0143] By designing the interactive process of multi-round dialogues, it is convenient to simulate the actual multi-round generation of data, further generate richer training data, and further improve the robustness and context understanding ability of the language model.

[0144] After self-supervised training of the language model on a large amount of corpus, fine-tuning of the backend tasks allows for more flexible configuration of intent lists without the need to retrain the language model, resulting in higher training efficiency and improved robustness and contextual understanding capabilities of the language model, while balancing the rapid recall of simple commands with the intelligence of the voice assistant.

[0145] In one embodiment, after the step of determining whether there is a human voice based on the audio segment of the wake-up word, the method further includes:

[0146] Mapping the audio segment of the wake-up word containing human voice to multi-channel audio data;

[0147] Determine the direction of the wake-up voice on the multi-channel audio data using an offline DOA algorithm to obtain the azimuth of the wake-up voice;

[0148] Based on the azimuth of the wake-up voice, a beamforming weight vector is constructed, a minimum variance distortionless response beamformer is applied to perform beamforming, and array output power variance is obtained;

[0149] Minimizing the power variance so that the response of the beamformer in the azimuth direction of the wake-up voice is maximized and the response in other directions is minimized, thereby obtaining preprocessed audio;

[0150] The mask of the 3A model is reused on the preprocessed audio to obtain and update the audio segment of the wake-up word containing human voice.

[0151] Through an offline DOA (Direction of Arrival) algorithm, the audio segments containing the wake-up word containing human voices are mapped to multi-channel audio data, obtaining the approximate azimuth of the wake-up person, enabling more accurate judgment of the angle of the wake-up person's sound source. Once the wake-up person's speaking angle is known, the minimum variance distortionless response (MVDR) beamformer is guided to perform beamforming, minimizing interfering sound sources from other angles while retaining the wake-up person's voice and achieving a sound barrier. At the same time, the mask generated by the 3A model in the front-end of the audio multiplexing after MVDR processing is reused to obtain very clean audio data, greatly suppressing human voices from other angles. This enables more appropriate processing of multi-channel voice data to better identify human voice commands, further improving recognition accuracy. Reusing the mask of the 3A model can save computing performance and eliminate the need for additional noise reduction processing, making audio noise reduction processing more convenient.

[0152] MVDR can also be replaced with a multi-channel sound barrier technology based on deep learning to suppress human voices from other angles to a greater extent. I will not go into details here.

[0153] In summary, a speech recognition method for microphone array conferencing equipment suppresses echo, noise and reverberation through an intelligent front-end 3A algorithm, thereby improving voice quality and the accuracy of voice wake-up and command recognition; at the same time, a more flexible voice wake-up recognition model is designed, which can flexibly configure more voice wake-up commands and response templates; and through array beamforming technology and voiceprint recognition technology, the recognition rate of human voice commands is improved; a more complete natural language understanding parser nlu parser (language model) is designed. As a voice assistant for microphone array conferencing equipment, it has better voice quality in complex environments such as conference rooms with strong reverberation and a large number of people, can more accurately recognize voice wake-up and response, and has more intelligent voice command parsing.

[0154] The 3A model, voice wake-up model, and voice recognition model all use grouped LSTM layers and grouped fully connected layers, which optimize computing performance and facilitate implementation on embedded devices.

[0155] A speech recognition method for microphone array conferencing equipment designs a full link of voice assistants from collecting single-channel audio from multi-channel conferencing equipment to command recognition. It can process multi-channel audio more reasonably and eliminate many redundant operations. The joint design of DOA+Mvdr and voiceprint recognition (or sound barrier)+VAD modules can greatly eliminate the interference of other speakers in conference room scenarios, making speech recognition more accurate.

[0156] A speech recognition method for microphone array conferencing equipment designs a set of fast matching algorithms and speech model implementation methods to achieve rapid response to simple commands and understand complex instruction expressions. At the same time, the language model designs a more convenient multi-round dialogue method, making it easier to understand the context.

[0157] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0158] Reference Figure 6 The embodiment of the present application further provides a speech recognition device for a microphone array conference device, which corresponds one-to-one to the speech recognition method for a microphone array conference device in the above embodiment. The speech recognition device for a microphone array conference device includes:

[0159] The acquisition module is used to obtain the speech data to be recognized in any single channel to obtain the input audio;

[0160] a noise reduction module, configured to perform noise reduction processing on the input audio using a preset 3A model to obtain target audio; the 3A model performs frequency band modeling on the input audio, fuses the extracted features, and calculates a mask through a complex network; the mask is applied to the input audio to obtain the target audio;

[0161] A wake-up word detection module is used to input the target audio into a preset voice wake-up model, output whether a wake-up word exists and the wake-up rate of the wake-up word, and determine the audio segment of the wake-up word;

[0162] A human voice detection module is used to determine whether there is a human voice based on the audio segment of the wake-up word;

[0163] A voiceprint matching module is used to extract the wake-up person's voiceprint features in the audio segment of the wake-up word containing the human voice using a preset extraction model when a human voice is present, determine whether the current frame is the wake-up person's voice, and determine the audio segment of the wake-up person;

[0164] The human voice command recognition module is used to input the audio segment of the current frame for waking up the person into a preset human voice recognition model for human voice command recognition, obtain the human voice command for waking up the person, identify the intention corresponding to the human voice command through the language model, and respond to the human voice command for waking up the person.

[0165] Mapping the audio segment of the wake-up word containing human voice to multi-channel audio data;

[0166] A speech recognition device for a microphone array conference device further includes:

[0167] The wake-up voice orientation enhancement module is used to determine the orientation of the wake-up voice on the multi-channel audio data through an offline DOA algorithm to obtain the azimuth of the wake-up voice; based on the azimuth of the wake-up voice, a beamforming weight vector is constructed, and a minimum variance distortionless response beamformer is applied to perform beamforming, and array output power variance; the power variance is minimized so that the response of the beamformer in the azimuth direction of the wake-up voice is maximized, and the response in other directions is minimized to obtain preprocessed audio; the mask of the 3A model is reused on the preprocessed audio to obtain and update the audio segment of the wake-up word with human voice.

[0168] For the specific definition of a speech recognition device for a microphone array conference device, please refer to the above definition of a speech recognition method for a microphone array conference device, which will not be repeated here.

[0169] Each module in the aforementioned speech recognition device for a microphone array conferencing system can be implemented in whole or in part via software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor within a computer device in hardware form, or can be stored in a computer device memory in software form, allowing the processor to call and execute the corresponding operations of each module.

[0170] In one embodiment, a computer device is provided, which may be a server. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements any of the above-mentioned speech recognition methods for a microphone array conferencing device.

[0171] In one embodiment, a computer-readable storage medium is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any one of the above-mentioned speech recognition methods for a microphone array conferencing device is implemented.

[0172] In one embodiment, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, any one of the above-mentioned speech recognition methods for a microphone array conferencing device is implemented.

[0173] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. When the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0174] Those skilled in the art will clearly understand that for the sake of convenience and brevity in description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A speech recognition method for a microphone array conference device, characterized in that: The following steps are included: Get any single channel of speech data to be recognized and get the input audio; Using a preset 3A model to perform noise reduction on the input audio to obtain a target audio; The 3A model performs frequency band modeling on the input audio, fuses the extracted features, and calculates them through a complex network to obtain a mask, which is applied to the input audio to obtain the target audio; Input the target audio into a preset voice wake-up model, output whether a wake-up word exists and the wake-up rate of the wake-up word, and determine the audio segment of the wake-up word; Determine whether there is a human voice based on the audio segment of the wake-up word; If there is a human voice, a preset extraction model is used to extract the wake-up voiceprint features in the audio segment of the wake-up word in which the human voice is present, to determine whether the current frame is the voice of the wake-up person, and to determine the audio segment of the wake-up person; The audio segment of the person being awakened in the current frame is input into a preset human voice recognition model for human voice command recognition to obtain the human voice command for waking up the person, the intention corresponding to the human voice command is identified through the language model, and the human voice command for waking up the person is responded to; the human voice recognition model includes a streaming inference model and a non-streaming inference model, and the recognition process of the human voice recognition model includes: model initialization, designing the corresponding keywords in the preset entity slot information list into hot words, and loading them into the non-streaming inference model; the audio segment of the person being awakened in the current frame is input into the streaming inference model for real-time speech recognition, and outputting real-time text data; if the audio segment of the person being awakened in the current frame is the last frame, all the previous audio segments of the person being awakened are integrated, input into the non-streaming inference model for recognition, reasoning and search, and output complete text data; Among them, it also includes: according to a preconfigured intent list, a finite state machine is constructed based on the slots and entity information in the intent list, and whether corresponding entity slots exist in the real-time text data and the complete text data; at the same time, if corresponding entity slots exist, a search tree is constructed according to the entity slots; the entity slots are used to search the search tree in a depth-first manner to obtain the corresponding intent and the required slot information and return them, and at the same time respond to the voice command of the wake-up person.

2. The speech recognition method for microphone array conference equipment according to claim 1, characterized in that: The 3A model includes two Conv2d layers, one LSTM layer and one concat layer; An input end of one of the Conv2d layers receives the signal after linear echo cancellation, and an output end is connected to the input end of the concat layer; An input end of another Conv2d layer receives a linearly predicted echo signal and is connected to an input end of the concat layer via the LSTM layer; The 3A model also includes three groups of transfer modules consisting of an LSTM layer, an LN layer and a Bi-LSTM layer connected in series in sequence. The three groups of transfer modules are arranged in series, and the output end of the concat layer is connected to the input end of the first group of transfer modules.

3. The speech recognition method for microphone array conference equipment according to claim 1, characterized in that: The 3A model training includes the following steps: A complex spectrum compression loss function is used for model training. By designing the compression rate of the complex spectrum compression loss function and the loss ratios of echo suppression, noise suppression and near-end sound quality, echo suppression, noise reduction and dereverberation of the target audio are performed.

4. The speech recognition method for microphone array conference equipment according to claim 1, characterized in that: The voice wake-up model includes: 3 translation modules consisting of a Transformer layer connected in series with an LSTM layer, a CTC fully connected layer, and a softmax layer; The three groups of translation modules are arranged in series, and the output end of the last group of translation modules is connected to the input end of the CTC fully connected layer, and the output end of the CTC fully connected layer is connected to the input end of the softmax layer.

5. The speech recognition method for microphone array conference equipment according to claim 1, characterized in that: The extraction model includes: Two groups of extraction modules are composed of a Conv2d layer, a Bi-LSTM layer, and a Pool layer connected in series. The input end of the first group of extraction modules receives the FBANK features in the audio segment of the wake-up word, and the output end of the second group of extraction modules outputs the voiceprint features of the wake-up person.

6. The speech recognition method for microphone array conference equipment according to claim 1, characterized in that: Before the step of querying whether there are corresponding physical slots in the real-time text data and the complete text data, the method further includes: The Chinese numeral expressions representing quantities in the real-time text data and the complete text data are converted into Arabic numerals, and modal particles in the real-time text data and the complete text data are filtered.

7. The speech recognition method for microphone array conference equipment according to any one of claims 1 to 6, characterized in that: After the step of determining whether there is a human voice according to the audio segment of the wake-up word, the method further includes: Mapping the audio segment of the wake-up word containing human voice to multi-channel audio data; Determine the direction of the wake-up voice on the multi-channel audio data using an offline DOA algorithm to obtain the azimuth of the wake-up voice; Based on the azimuth of the wake-up voice, a beamforming weight vector is constructed, a minimum variance distortionless response beamformer is applied to perform beamforming, and array output power variance is obtained; Minimizing the power variance so that the response of the beamformer in the azimuth direction of the wake-up voice is maximized and the response in other directions is minimized, thereby obtaining preprocessed audio; The mask of the 3A model is reused on the preprocessed audio to obtain and update the audio segment of the wake-up word containing human voice.

8. A speech recognition device for microphone array conference equipment, characterized in that: Used to execute the speech recognition method for a microphone array conference device according to any one of claims 1 to 7, include, The acquisition module is used to obtain the speech data to be recognized in any single channel to obtain the input audio; A noise reduction module, configured to perform noise reduction processing on the input audio using a preset 3A model to obtain a target audio; The 3A model performs frequency band modeling on the input audio, fuses the extracted features, and calculates them through a complex network to obtain a mask, which is applied to the input audio to obtain the target audio; A wake-up word detection module is used to input the target audio into a preset voice wake-up model, output whether a wake-up word exists and the wake-up rate of the wake-up word, and determine the audio segment of the wake-up word; A human voice detection module is used to determine whether there is a human voice based on the audio segment of the wake-up word; A voiceprint matching module is used to extract the wake-up person's voiceprint features in the audio segment of the wake-up word containing the human voice using a preset extraction model when a human voice is present, determine whether the current frame is the wake-up person's voice, and determine the audio segment of the wake-up person; The human voice command recognition module is used to input the audio segment of the current frame for waking up the person into a preset human voice recognition model for human voice command recognition, obtain the human voice command for waking up the person, identify the intention corresponding to the human voice command through the language model, and respond to the human voice command for waking up the person.

Images