Whisper audio transcription method, system, and storage medium based on hot word motivation

Abstract

The present invention discloses a Whisper audio transcription method, system, and storage medium based on hot word excitation. The present invention proposes a hot word framework, adds a bias encoder / decoder module to the Whisper base framework, and modifies a part of the calculation logic. Among them, the bias encoder is used to extract hot word text features, and the bias decoder combines the hot word feature excitation to transcribe the text, so that the keywords that may have been transcribed incorrectly converge in the direction of the hot word feature. Through hot word excitation, Whisper corrects the erroneous text and obtains the correct text content. Similarly, when an abnormal voice transcription occurs on site, the transcription error correction can be quickly achieved by retaining / inputting hot words, thereby optimizing the fluency of human-computer interaction.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and more specifically, to a Whisper audio transcription method, system, and storage medium based on hotword excitation. Background Art

[0002] In industrial production environments, operators often need to query production data or related information during operations. Existing query tools require operators to manually operate on the query device. When the operator is operating other equipment, the operator needs to switch equipment, resulting in the operator having to leave the machine work when querying production data or paging, which cannot maximize production efficiency.

[0003] Currently, speech recognition model architectures are mainly divided into end-to-end (represented by Conformer) and autoregressive (represented by Whisper) types. The end-to-end framework has a fast inference speed, while the autoregressive framework has a low word error rate (WER) and low perplexity level (PPL). Although existing speech recognition and search technologies are relatively mature, the accuracy of existing speech recognition is low for proprietary vocabulary in specific industrial fields, which makes existing speech recognition and search functions difficult to use in specific industrial fields. In view of the pain points in the above industrial fields, the selection of speech recognition frameworks has become very important. Obviously, higher speech recognition accuracy and better industrial hot word correction capabilities are more suitable for industrial production fields. Summary of the Invention

[0004] The present invention overcomes the shortcomings of the prior art in that speech recognition accuracy is low in specific fields, and provides a Whisper audio transcription method, system, and storage medium based on hot word excitation, in the hope of solving the problems existing in the prior art.

[0005] To solve the above technical problems, one aspect of the present invention provides a Whisper audio transcription method based on hotword motivation, comprising the following steps:

[0006] S1: obtain audio signal;

[0007] S2: Pre-emphasize the signal by frame windowing, use fast Fourier transform to obtain the signal spectrum, and filter the spectrum with triangular filter to obtain the Mel spectrum;

[0008] S3: Convolve the obtained Mel spectrum to extract the underlying audio features and perform time compression;

[0009] S4: After several layers of transformer encoder layer and several layers of decoder layer;

[0010] S5: introduce bias encoder and bias decoder modules;

[0011] S6: Hot word text is input into bias encoder to obtain hot word text features;

[0012] S7: The Mel spectrum is input into Whisper-Encoder to obtain enhanced spectrum features;

[0013] S8: The text start symbol is combined with the spectrum feature and input into Whisper-Decoder to obtain the candidate text feature;

[0014] S9: The hot word text features and candidate text features are input into the bias decoder, and the transcribed text is output.

[0015] The present invention adds a bias encoder / decoder module to the Whisper base framework and modifies part of the calculation logic. The bias encoder is used to extract hot word text features, and the bias decoder combines hot word features to stimulate the transcription of text, so that keywords that may have been transcribed incorrectly converge in the direction of hot word features. Through hot word stimulation, Whisper corrects erroneous text and obtains the correct text content. Similarly, when voice transcription anomalies occur on site, transcription error correction can be quickly achieved by retaining / inputting hot words, thereby optimizing the fluency of human-computer interaction.

[0016] A further technical solution is that S2 includes the following steps:

[0017] S21: Pre-emphasizes the audio signal and performs windowing to prevent spectrum leakage;

[0018] S22: Perform fast Fourier transform to obtain audio spectrum;

[0019] S23: Use a triangular filter to filter out impurity frequencies in the spectrum characteristics.

[0020] A further technical solution is that the S9 includes the following steps:

[0021] S91: Combine the text feature X to be transcribed, the hot word feature Y and the parameter matrix W q / W k / W v Through linear operations on the matrices, we obtain the query matrix Q representing the transcribed text, the key matrix K representing the hot words, and the value matrix V representing the hot words;

[0022] The above Q, K, and V represent (query, key, value), which are traditional variables in multi head attention.

[0023] S92: Perform linear operations on the matrix Q and the matrix K to obtain the attention score;

[0024] S93: Perform a linear operation on the attention score and the matrix V to obtain the output of the same shape as X hw ;

[0025] S94: Weighted output hw And obtain the text prediction feature output through the original cross attenton module to obtain the predicted transcript.

[0026] A further technical solution is that the merging in S94 specifically includes the following steps:

[0027] S941: Determine the hot word weight parameter a of the unit shape;

[0028] S942: Multiply the output outputhw by the hot word weight parameter a, and then merge it with the text prediction feature output to obtain the predicted transcription text.

[0029] A further technical solution is to perform the following steps before S5:

[0030] S50: Replace the linear perception layer of Whisper-Decoder with a causal convolutional perception layer;

[0031] S51: Set the one-dimensional convolution kernel (conv) size ksize;

[0032] S52: Calculate the features of the current token using the current and previous ksize-1 tokens.

[0033] A further technical solution is to perform the following steps after S51:

[0034] S510: Determine the size of the casual padding and perform the casual padding to ensure that the left padding size of the text feature is always ksize-1 and the right padding size is always 1.

[0035] Similar to the traditional Transformer, the linear perception layer of Whisper-Decoder adopts a causal mask (Causal Mask) and a Causal Mask structure. However, the causal mask ensures that the prediction of each token depends only on the current position and the token before it, but not on subsequent tokens. In order to enhance Whisper's ability to understand the context, the present invention replaces the linear perception layer (MLP) of Whisper-Decoder with a causal convolutional perception layer (Causal Convolutional Layer, CCL). Specifically, by setting the one-dimensional convolution kernel (conv) size ksize, the calculation of the token depends on the current and previous ksize-1 tokens, thereby enhancing the context capability.

[0036] The present invention also provides a Whisper audio transcription system based on hot word excitation, including an audio acquisition module, a spectrum conversion module, an encoder, a decoder, and a bias encoder / decoder module;

[0037] The audio acquisition module is used to obtain audio signals;

[0038] The spectrum conversion module is used to convert the audio signal into a Mel spectrum;

[0039] The encoder is used to enhance audio features;

[0040] The decoder is used to map features to text features, enhance text representation, and achieve accurate transcription of text;

[0041] In the bias encoder / decoder module, the bias encoder is used to extract the hot word text features, and the bias decoder is used to combine the hot word features to stimulate the transcription of text, so that the keywords that may have been transcribed incorrectly converge towards the hot word features.

[0042] A further technical solution is that the audio transcription system also includes a causal convolutional perception layer;

[0043] The causal convolutional perception layer is used for token calculation.

[0044] The present invention also provides a computer-readable storage medium comprising computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to perform the speech recognition method as described above.

[0045] Compared with the prior art, the present invention has at least the following beneficial effects: Whisper has difficulty in accurately identifying uncommon words or professional terms in niche areas, and the transcribed text may have the same pronunciation but incorrect content. The present invention proposes a hot word framework, adds a bias encoder / decoder module to the Whisper base framework, and modifies part of the calculation logic. Among them, the bias encoder is used to extract hot word text features, and the bias decoder combines the hot word features to stimulate the transcribed text, so that the keywords that may have been transcribed incorrectly converge in the direction of the hot word features. Through hot word stimulation, Whisper corrects the erroneous text and obtains the correct text content. Similarly, when voice transcription anomalies occur on site, transcription error correction can be quickly achieved by retaining / inputting hot words, thereby optimizing the fluency of human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a schematic diagram of the model architecture of the first embodiment of the present invention;

[0047] Figure 2 Schematic diagram of the structure and calculation principle of bias decoder;

[0048] Figure 3 This is a schematic diagram of the attention score table scores;

[0049] Figure 4 Schematic diagram of the linear perception layer structure;

[0050] Figure 5 Schematic diagram of the causal convolutional perception layer structure. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0052] Example 1

[0053] A Whisper audio transcription method based on hotword motivation includes the following steps:

[0054] S1: obtain audio signal;

[0055] S2: Pre-emphasize the signal by frame windowing, use fast Fourier transform to obtain the signal spectrum, and filter the spectrum with triangular filter to obtain the Mel spectrum;

[0056] In this embodiment, S2 includes the following steps:

[0057] S21: Pre-emphasizes the audio signal and performs windowing to prevent spectrum leakage;

[0058] S22: Perform fast Fourier transform to obtain audio spectrum;

[0059] S23: Use a triangular filter to filter out impurity frequencies in the spectrum characteristics.

[0060] S3: Convolve the obtained Mel spectrum to extract the underlying audio features and perform time compression;

[0061] S4: After several layers of transformer encoder layer and several layers of decoder layer;

[0062] S5: introduce bias encoder and bias decoder modules;

[0063] It should be noted that the bias encoder is a single-layer long short-term memory module (Long Short-Term Memory, LSTM).

[0064] S6: Hot word text is input into bias encoder to obtain hot word text features;

[0065] See also Figure 1 When the user provides a hot word for the manufacturing field, the hot word features are extracted:

[0066]

[0067] Each hot word is represented as a hot word feature to facilitate the subsequent finding of the connection between the hot word and the transcribed text.

[0068] S7: The Mel spectrum is input into Whisper-Encoder to obtain enhanced spectrum features;

[0069] S8: The text start symbol is combined with the spectrum feature and input into Whisper-Decoder to obtain the candidate text feature;

[0070] In S8, the autoregressive model uses the start symbol <s>As the starting input for text prediction, it is a common practice. Candidate text refers to the features output by traditional whisper. Because it has not yet been combined with hot words, it is called candidate features.

[0071] S9: The hot word text features and candidate text features are input into the bias decoder, and the transcribed text is output.

[0072] By improving the cross-attention layer in Whisper-decoder, the transcribed text features focus on spectral features while also being motivated by hot words.

[0073] Further preferably, the S9 includes the following steps:

[0074] S91: Combine the text feature X to be transcribed, the hot word feature Y and the parameter matrix W q / W k / W v Through linear operations on the matrices, we obtain the query matrix Q representing the transcribed text, the key matrix K representing the hot words, and the value matrix V representing the hot words;

[0075] The bias decoder consists of a layer of cross attention module. See the structure for details. Figure 2 , where X represents the text feature to be transcribed; Y represents the hot word feature; Q / K / V are respectively composed of X / Y and the parameter matrix W q / W k / W v It is obtained through linear operations of the matrix. The calculation steps are as follows:

[0076]

[0077] Among them, the shape of X is (n, d), n represents the length of the text; the shape of Y is (m, d), m represents the number of hot words; W q / W k / W v / W o The shape is (d, d); d represents the number of channels. The matrix Q (represents the transcript) and K (represents the hot words);

[0078] S92: Perform linear operations on the matrix Q and the matrix K to obtain the attention score;

[0079] Perform linear operations to get the attention score:

[0080]

[0081] where K T Represents the transpose of matrix K, with a shape of (d, m); the softmax activation function constrains all elements of the matrix to be between (0, 1) and is used to represent the score. The calculation steps can be found in Figure 2 .

[0082] The shape of scores is (n,m), and its representation is shown in Figure 3 ;

[0083] exist Figure 3 In the example, when the transcribed text is more similar to the hot word, the darker the color of the attention position (the higher the score), it is easier to stimulate the transcribed text and enable the model to infer the correct text content; conversely, when the transcribed text is unrelated to the hot word, the lighter the color of the attention position (the lower the score), it will not affect the text content inferred by the model.

[0084] S93: Perform a linear operation on the attention score and the matrix V to obtain the output of the same shape as X hw ;

[0085] The principle is to count how many points the original text gets in the hot word text, and take as many points from v as the original text gets, and the more relevant the text, the higher the score.

[0086] S94: Weighted output hw And obtain the text prediction feature output through the original cross attenton module to obtain the predicted transcript.

[0087] Perform linear operations on scores and matrix V (representing hot words):

[0088] output hw =(score@V)@W o

[0089] Get the output of the same shape as X hw , which has a representation of hot words, called hot word prediction features. In the Whisper base model, text features and spectral features are also output through the original cross attenton module to obtain text prediction features.

[0090] The merging in S94 specifically includes the following steps:

[0091] S941: Determine the hot word weight parameter a of the unit shape;

[0092] S942: Multiply the output outputhw by the hot word weight parameter a, and then merge it with the text prediction feature output to obtain the predicted transcription text.

[0093] Merge the two:

[0094] output=output+α·output hw

[0095] Here, α is a unit-shaped parameter between (0, 1) and is used to control the proportion of hot word features. Finally, the output is input into the CCL module to obtain the predicted transcript.

[0096] The Whisper-decoder consists of n sub-modules: a self-attention layer, a cross-attention layer, and a linear perception layer. A bias decoder module is introduced in the last sub-module of the Whisper-decoder to control the transcribed text from being stimulated by hot words and improve the recognition rate of proper nouns.

[0097] In a further preferred embodiment, the following steps are performed before S5:

[0098] S50: Replace the linear perception layer of Whisper-decoder with a causal convolutional perception layer;

[0099] To enhance Whisper's ability to understand the context, this solution replaces the Whisper-decoder's linear perception layer (MLP) with a causal convolutional perception layer (CCL). Figure 4 , each token in MLP performs parallel computation without affecting each other. The causal convolution perception layer structure can be found in Figure 5 The principle is to set the one-dimensional convolution kernel (conv) size ksize so that the calculation of the token depends on the current and previous ksize-1 tokens, thereby enhancing the context capability.

[0100] For example, the specific steps of MLP→CCL calculation conversion are:

[0101] S51: Set the one-dimensional convolution kernel size ksize;

[0102]

[0103] Where linear / conv represent the linear / convolution operation of the matrix respectively; activate represents the activation function excitation feature;

[0104] S510: Determine the size of the causal padding and perform the causal padding to ensure that the left padding size of the text feature is always ksize-1 and the right padding size is always 1.

[0105] Since causal convolution is related to the above, causal padding is required to avoid information loss. When causal padding meets the conditions:

[0106]

[0107] Causal padding ensures that the length of the token sequence after causal convolution is not affected. ksize directly determines the number of tokens mentioned above, while stride represents the convolution step size.

[0108] S52: Calculate the features of the current token using the current and previous ksize-1 tokens.

[0109] Through hot word incentives, Whisper can correct incorrect text and obtain the correct text content.

[0110]

[0111] Example 2

[0112] A Whisper audio transcription system based on hotword excitation, including an audio acquisition module, a spectrum conversion module, an encoder, a decoder, a bias encoder / decoder module, and a causal convolutional perception layer;

[0113] The audio acquisition module is used to obtain audio signals;

[0114] The spectrum conversion module is used to convert the audio signal into a Mel spectrum;

[0115] The encoder is used to enhance audio features;

[0116] The decoder is used to map features to text features, enhance text representation, and achieve accurate transcription of text;

[0117] In the bias encoder / decoder module, the bias encoder is used to extract the hot word text features, and the bias decoder is used to combine the hot word features to stimulate the transcription of text, so that the keywords that may have been transcribed incorrectly converge towards the hot word features.

[0118] The causal convolutional perception layer is used for token calculation.

[0119] Example 3

[0120] A computer-readable storage medium is characterized by comprising computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to perform the speech recognition method as described above.

[0121] Although the present invention has been described herein with reference to illustrative embodiments of the present invention, it will be appreciated that those skilled in the art may devise numerous other modifications and implementations that fall within the scope and spirit of the principles disclosed herein. More specifically, within the scope disclosed herein, various variations and improvements may be made to the components and / or layout of the subject combination layout. In addition to variations and improvements made to the components and / or layout, other uses will be apparent to those skilled in the art.< / s>

Claims

1. A Whisper audio transcription method based on hot word excitation, characterized in that: The following steps are involved: S1: obtain audio signal; S2: Pre-emphasize the signal by frame windowing, use fast Fourier transform to get the signal spectrum, and filter the spectrum with triangular filter to get Mel spectrum; S3: Convolve the obtained Mel spectrum to extract the underlying audio features and compress them in time series; S4: After several layers of transformer encoder layer and several layers of decoder layer; S5: introduce bias encoder and bias decoder modules; S6: The hot word text is input into the bias encoder to obtain the hot word text features; S7: Mel spectrum is input into Whisper-Encoder to obtain enhanced spectrum features; S8: The text start symbol is combined with the spectrum feature and input into Whisper-Decoder to obtain the candidate text feature; S9: The hot word text features and candidate text features are input into the bias decoder, and the transcribed text is output.

2. A Whisper audio transcription method based on hot word motivation as claimed in claim 1, characterized in that: The S2 comprises the following steps: S21: pre-emphasize the audio signal and perform windowing to prevent spectrum leakage; S22: Perform fast Fourier transform to obtain an audio spectrum; S23: Use a triangular filter to filter out impurity frequencies in the spectrum features.

3. A Whisper audio transcription method based on hot word motivation as claimed in claim 1, characterized in that: The S9 comprises the following steps: S91: Combine the text feature X to be transcribed, the hot word feature Y and the parameter matrix W q / W k / W v Through linear operations on matrices, we obtain the query matrix Q representing the transcribed text, the key matrix K representing the hot words, and the value matrix V representing the hot words; S92: Perform linear operations on the matrix Q and the matrix K to obtain an attention score; S93: Perform a linear operation on the attention score and matrix V to obtain an output of the same shape as X hw ; S94: Weighted output hw And get the text prediction feature output through the original cross attenton module to get the predicted transcription text.

4. A Whisper audio transcription method based on hot word motivation as described in claim 3, characterized in that: The merging in S94 specifically includes the following steps: S941: Determine the hot word weight parameter a of the unit shape; S942: Output hw Multiply it with the hot word weight parameter a, and then merge it with the text prediction feature output to get the predicted transcription text.

5. A Whisper audio transcription method based on hot word motivation as claimed in claim 1, characterized in that: Perform the following steps before S5: S50: Replace the linear perception layer of Whisper-Decoder with a causal convolutional perception layer; S51: Set the one-dimensional convolution kernel size ksize; S52: Use the current and previous ksize-1 tokens to calculate the features of the current token.

6. A Whisper audio transcription method based on hot word motivation as claimed in claim 5, characterized in that: After S51, perform the following steps: S510: Determine the size of the causal padding and perform the causal padding to ensure that the left padding size of the text feature is always ksize-1 and the right padding size is always 1.

7. A Whisper audio transcription system based on hot word motivation, characterized in that: Including audio acquisition module, spectrum conversion module, encoder, decoder, bias encoder / decoder module; The audio acquisition module is used to obtain audio signals; The spectrum conversion module is used to convert the audio signal into a Mel spectrum; The encoder is used to enhance audio features; The decoder is used to map features to text features, enhance text representation, and achieve accurate transcription of text; In the bias encoder / decoder module, the bias encoder is used to extract the hot word text features, and the bias decoder is used to combine the hot word features to stimulate the transcription of the text, so that the keywords that may have been transcribed incorrectly converge to the hot word features.

8. A Whisper audio transcription system based on hot word motivation as claimed in claim 7, characterized in that: Also includes,causal convolutional perception layer; The causal convolutional perception layer is used for token calculation.

9. A computer-readable storage medium, characterized in that: The invention comprises computer executable instructions, which, when executed by one or more processors, enable the one or more processors to perform the speech recognition method according to any one of claims 1 to 6.

Images