Application of VAD method based on deep learning in speech recognition system

Abstract

The invention relates to the technical field of voice recognition, and discloses a voice recognition method applied to a VAD module, and the method comprises the steps: taking a frame as a unit, obtaining audio stream data, and carrying out the up-sampling or down-sampling of the audio stream data into a needed format; performing feature extraction on each frame of audio stream data through a multi-head attention model to obtain feature data of each frame of audio stream data; setting a sliding window to record feature data of the multi-frame audio stream data; overall judgment of a speaking state or other states is carried out on the feature data in the single sliding window, and audio stream data judged to be in other states are removed; the voice-to-character conversion accuracy is effectively improved, the computing resource consumption is reduced, and the voice-to-character conversion efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and specifically to a speech recognition method applied to a VAD module. Background Art

[0002] The VAD module currently used in the speech recognition system is based on the open source project WebRTC, which is gradually unable to meet the accuracy requirements of the speech recognition system. The accuracy of audio endpoint detection is low, and traditional methods can basically not distinguish between noise and human voices, which brings interference to the subsequent speech-to-text processing.

[0003] Application Contents

[0004] The purpose of this application is to provide a speech recognition method applied to a VAD module that effectively improves the accuracy of speech-to-text conversion, reduces computing resource consumption, and improves the conversion efficiency of speech-to-text.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A speech recognition method applied to a VAD module of the present invention comprises acquiring audio stream data in units of frames, and upsampling or downsampling the audio stream data into a required format;

[0007] The multi-head attention model is used to extract features from each frame of audio stream data to obtain feature data of each frame of audio stream data;

[0008] Set a sliding window to record feature data of multiple frames of audio stream data;

[0009] An overall judgment of the speaking state or other state is made on the feature data in a single sliding window, and the audio stream data judged to be in other states is discarded.

[0010] Further, the method of extracting features of each frame of audio stream data through a multi-head attention model to obtain feature data of each frame of audio stream data includes:

[0011] The audio stream data that has been upsampled or downsampled is subjected to short-time Fourier transform to extract the frequency domain information of each frame of audio stream data;

[0012] The frequency domain information of each frame of audio stream data is input into the neural network, the probability that each frame of audio stream data is a human voice signal is calculated, and output as feature data to achieve feature extraction.

[0013] Furthermore, the method of making an overall judgment of the speaking state or other state on the feature data in a single sliding window and removing the audio stream data judged to be in other states includes: performing weighted calculation on the feature data in the sliding window to obtain the probability that the feature data in the single sliding window is in a human voice state;

[0014] Compare the probability that the feature data in the sliding window is a human voice state with a first threshold or a second threshold, wherein the first threshold is greater than the second threshold;

[0015] If the probability that the feature data in the sliding window is in a human voice state is greater than a first threshold, all the multiple frames of audio stream data in the sliding window are judged as being in a speaking state and output;

[0016] If the probability that the feature data in the sliding window is in a human voice state is less than a second threshold, multiple frames of audio stream data in the sliding window are all judged as being in other states and are discarded;

[0017] If the probability that the feature data in the sliding window is in a human voice state is greater than or equal to the second threshold and less than or equal to the first threshold, the multiple frames of audio stream data in the sliding window are all judged to be in the same state as the previous sliding window and are output or discarded.

[0018] Furthermore, when the feature data of N consecutive frames of audio stream data in the sliding window are all greater than a first threshold, the audio stream data in the sliding window are all in a speaking state and are output.

[0019] Compared with the prior art, the beneficial effects of this application are:

[0020] In practical applications, by upsampling or downsampling the audio stream data, preprocessing of each frame of audio stream data is achieved, and the preprocessed audio stream data is input into a multi-head attention model. Through the multi-head attention model, feature extraction is performed on each frame of audio stream data to obtain feature data of each frame of audio stream data, and a sliding window is set to record the feature data of multiple frames of audio stream data. When there are N consecutive frames of audio stream data in the sliding window in a human voice state, the multiple frames of audio stream data recorded in the sliding window are all output in a speaking state. On the premise of ensuring recognition accuracy, the consumption of computing resources is effectively reduced and the conversion efficiency of speech to text is improved; otherwise, the audio stream data in the sliding window is judged as a speaking state or other state, and the audio stream data judged as other states is eliminated to achieve noise filtering, thereby further improving the accuracy of speech to text; the present application effectively improves the accuracy of speech to text, reduces the consumption of computing resources, and improves the conversion efficiency of speech to text. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the implementation methods of the present application, the drawings required for use in the implementation methods will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a flow chart of the method of this application. DETAILED DESCRIPTION

[0023] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0024] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show the layers related to the present invention rather than being drawn according to the number, shape and size of the layers in actual implementation. In actual implementation, the type, quantity and proportion of each layer may be changed arbitrarily, and the layer layout may also be more complicated.

[0025] In the following description, numerous details are discussed to provide a more thorough explanation of embodiments of the present invention; however, it is apparent to one skilled in the art that embodiments of the present invention may be practiced without these specific details.

[0026] See also Figure 1 , a speech recognition method applied to a VAD module, comprising acquiring audio stream data in units of frames, and upsampling or downsampling the audio stream data into a required format;

[0027] The multi-head attention model is used to extract features from each frame of audio stream data to obtain feature data of each frame of audio stream data;

[0028] Set a sliding window to record feature data of multiple frames of audio stream data;

[0029] An overall judgment of the speaking state or other state is made on the feature data in a single sliding window, and the audio stream data judged to be in other states is discarded.

[0030] In practical applications, by upsampling or downsampling the audio stream data, preprocessing of each frame of audio stream data is achieved, and the preprocessed audio stream data is input into a multi-head attention model. Through the multi-head attention model, feature extraction is performed on each frame of audio stream data to obtain feature data of each frame of audio stream data, that is, the probability that each frame of audio stream data is in a human voice state, and a sliding window is set. This sliding window is a fixed-size area, which stores the feature data of the audio stream data in a certain time area. An overall judgment is made on the speaking state or other states of the feature data in a single sliding window, and audio stream data judged to be in other states is eliminated to achieve noise filtering, improve the accuracy of speech-to-text, and effectively reduce the consumption of computing resources, thereby improving the conversion efficiency of speech-to-text. The present application effectively improves the accuracy of speech-to-text, reduces the consumption of computing resources, and improves the conversion efficiency of speech-to-text.

[0031] The method of extracting features of each frame of audio stream data by a multi-head attention model to obtain feature data of each frame of audio stream data includes:

[0032] The audio stream data that has been upsampled or downsampled is subjected to short-time Fourier transform to extract the frequency domain information of each frame of audio stream data;

[0033] The frequency domain information of each frame of audio stream data is input into the neural network, the probability that each frame of audio stream data is a human voice signal is calculated, and output as feature data to achieve feature extraction.

[0034] By using formula 1, the input audio stream data is short-time Fourier transformed to obtain the frequency domain information of each frame of audio stream data:

[0035]

[0036] The frequency domain information of each frame of audio stream data is input into the neural network, and the representation vector of the frequency domain information of each frame of audio stream data is calculated by formulas 2 to 4:

[0037]

[0038] The similarity score between the representation vectors of frequency domain information is calculated by formula 5:

[0039]

[0040] The probability that each frame of audio stream data is a human voice signal is calculated using Formula 6 and Formula 7:

[0041]

[0042] in, is the output weight matrix; the multi-head attention model extracts features from each frame of audio stream data. Compared with CNN and RNN in complexity, the multi-head attention model has a smaller amount of calculation, reduces the consumption of computing resources, and improves the conversion efficiency of speech to text.

[0043] The method of judging whether the audio stream data in the sliding window is in a speaking state or other state and removing the audio stream data judged to be in other states comprises: performing weighted calculation on the feature data in the sliding window to obtain the probability that the feature data in a single sliding window is in a human voice state;

[0044] Compare the probability that the feature data in the sliding window is a human voice state with a first threshold or a second threshold, wherein the first threshold is greater than the second threshold;

[0045] If the probability that the feature data in the sliding window is in a human voice state is greater than a first threshold, the multiple frames of audio stream data in the sliding window are all output in a speaking state, and the audio stream data in the sliding window is judged in the next time period;

[0046] If the probability that the feature data in the sliding window is in a human voice state is less than a second threshold, multiple frames of audio stream data in the sliding window are all judged as being in other states and are discarded;

[0047] If the probability that the feature data in the sliding window is in a human voice state is greater than or equal to the second threshold and less than or equal to the first threshold, the multiple frames of audio stream data in the sliding window are all judged to be in the same state as the previous sliding window and are output or discarded.

[0048] The feature data of multiple frames of audio stream data recorded in a single sliding window are treated as a whole, and weighted calculation is performed on the status data of the multiple frames of audio stream data in the sliding window, and a proportional analysis is performed. Based on the proportional analysis results, the multiple frames of audio stream data recorded in the sliding window are recognized and judged as a whole, thereby further reducing the consumption of computing resources and improving the real-time performance of speech-to-text conversion.

[0049] When the characteristic data of N consecutive frames of audio stream data in the sliding window are all greater than the first threshold, the audio stream data in the sliding window are all in a speaking state and are output.

[0050] Wherein, N is an integer, and N is set to be greater than or equal to one half of the total number of frames of audio stream data in the sliding window; when the feature data of N consecutive frames of audio stream data in the sliding window are all greater than the first threshold, the audio stream data in the sliding window are all in a speaking state and output, which effectively reduces the consumption of computing resources and improves the conversion efficiency of speech to text while ensuring the recognition accuracy.

[0051] In the above-mentioned embodiments, although the present invention has been described in conjunction with the specific embodiments of the present invention, many replacements, modifications and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. The embodiments of the present invention are intended to cover all such replacements, modifications and variations falling within the broad scope of the appended claims.

[0052] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A speech recognition method applied to a VAD module, characterized in that: Including obtaining audio stream data in frames, and upsampling or downsampling the audio stream data into a required format; The multi-head attention model is used to extract features from each frame of audio stream data to obtain feature data of each frame of audio stream data; Set a sliding window to record feature data of multiple frames of audio stream data; An overall judgment of the speaking state or other state is made on the feature data in a single sliding window, and the audio stream data judged to be in other states is discarded.

2. A speech recognition method applied to a VAD module according to claim 1, characterized in that: The method of extracting features of each frame of audio stream data by a multi-head attention model to obtain feature data of each frame of audio stream data includes: The audio stream data that has been upsampled or downsampled is subjected to short-time Fourier transform to extract the frequency domain information of each frame of audio stream data; The frequency domain information of each frame of audio stream data is input into the neural network, the probability that each frame of audio stream data is a human voice signal is calculated, and output as feature data to achieve feature extraction.

3. A speech recognition method applied to a VAD module according to claim 1, characterized in that: The method for making an overall judgment of a speaking state or other state on feature data in a single sliding window and removing audio stream data judged to be in other states includes: performing weighted calculation on the feature data in the sliding window to obtain a probability that the feature data in the single sliding window is in a human voice state; Compare the probability that the feature data in the sliding window is a human voice state with a first threshold or a second threshold, wherein the first threshold is greater than the second threshold; If the probability that the feature data in the sliding window is in a human voice state is greater than a first threshold, all the multiple frames of audio stream data in the sliding window are judged as being in a speaking state and output; If the probability that the feature data in the sliding window is in a human voice state is less than a second threshold, multiple frames of audio stream data in the sliding window are all judged as being in other states and are discarded; If the probability that the feature data in the sliding window is in a human voice state is greater than or equal to the second threshold and less than or equal to the first threshold, the multiple frames of audio stream data in the sliding window are all judged to be in the same state as the previous sliding window and are output or discarded.

4. A speech recognition method applied to a VAD module according to claim 3, characterized in that: When the characteristic data of N consecutive frames of audio stream data in the sliding window are all greater than the first threshold, the audio stream data in the sliding window are all in a speaking state and are output.

Images