Chunking and duplicate decoding strategies for streaming RNN transformers for speech recognition

Abstract

A computer-implemented method for improving recognition accuracy of digital speech is provided. The method includes receiving the digital speech. The method further includes splitting the digital speech into overlapping chunks. The method also includes computing a bi-directional encoder embedding for each of the overlapping chunks to obtain a bi-directional encoder embedding. The method further includes combining the bi-directional encoder embeddings. The method further includes interpreting the digital speech using the combined bi-directional encoder embeddings by a speech recognition system.

Claims

1. A computer implementation method for improving the recognition accuracy of digital speech, Receiving the aforementioned digital audio, The digital audio is divided into multiple overlapping chunks, wherein two overlapping chunks adjacent to each other have overlapping regions. To obtain a bidirectional encoder embedding, the bidirectional encoder embedding is calculated for each of the multiple overlapping chunks, The aforementioned bidirectional encoder embedding is combined, A speech recognition system using a recurrent neural network converter model performs speech recognition of the digital speech using the coupled bidirectional encoder embedding, Computer implementation methods, including those mentioned above.

2. The computer implementation method according to claim 1, wherein the joining step, when joining the bidirectional encoder embeddings of a first overlapping chunk and a second overlapping chunk having overlapping regions, sets the bidirectional encoder embedding of the first overlapping chunk to zero in the overlapping region and joins the bidirectional encoder embedding of the second overlapping chunk with the zero.

3. The computer implementation method according to claim 1, wherein the joining step includes averaging the bidirectional encoder embedding of the first overlapping chunk and the bidirectional encoder embedding of the second overlapping chunk with the bidirectional encoder embedding of the second overlapping chunk in the overlapping region, when joining the bidirectional encoder embeddings of the first overlapping chunk and the second overlapping chunk having overlapping regions.

4. The computer implementation method according to claim 1, wherein the joining step includes, when joining the bidirectional encoder embeddings of two overlapping chunks having overlapping regions, using the bidirectional encoder embedding from the left of the overlapping chunks in the first half of the overlapping region and using the bidirectional encoder embedding from the right of the overlapping chunks in the second half of the overlapping region.

5. The computer implementation method according to claim 1, wherein the recurrent neural network converter model includes a joint network operably coupled to a prediction network and an encoder.

6. The computer implementation method according to claim 5, wherein the recurrent neural network converter model further comprises a softmax layer operably coupled to the joint network, which converts the output of the joint network into a conditional probability distribution.

7. The computer implementation method according to claim 1, wherein the duplicate chunks are used during the training session of the recurrent neural network converter model.

8. The computer implementation method according to claim 1, wherein the chunk size is randomized during the training session of the recurrent neural network converter model.

9. The computer implementation method according to claim 1, wherein the chunk size is fixed during the training session of the recurrent neural network converter model.

10. The computer implementation method according to claim 1, wherein the amount of chunk overlap is fixed during the training session of the recurrent neural network converter model.

11. The computer implementation method according to claim 1, wherein the amount of chunk overlap is randomized during the training session of the recurrent neural network converter model.

12. The computer implementation method according to claim 1, wherein the encoder is configured to simulate an acoustic model in the speech recognition system, and the prediction network is configured to simulate a language model in the speech recognition system.

13. The computer implementation method according to claim 1, wherein the speech recognition execution step includes performing a beam search on the alignment grid of the recurrent neural network converter model.

14. The computer implementation method according to claim 1, wherein the amount of overlap between the two overlapping chunks is an inference parameter used to control accuracy versus latency.

15. A computer program for improving the recognition accuracy of digital speech, wherein the computer program is executable by a computer, and the computer is made to execute a method, and the method is Receiving the aforementioned digital audio, The digital audio is divided into multiple overlapping chunks, wherein two overlapping chunks adjacent to each other have overlapping regions. To obtain a bidirectional encoder embedding, the bidirectional encoder embedding is calculated for each of the multiple overlapping chunks, The aforementioned bidirectional encoder embedding is combined, A speech recognition system using a recurrent neural network converter model performs speech recognition of the digital speech using the coupled bidirectional encoder embedding, A computer program that includes [this].

16. The computer program according to claim 15, wherein the joining step, when joining the bidirectional encoder embeddings of a first overlapping chunk and a second overlapping chunk having overlapping regions, sets the bidirectional encoder embedding of the first overlapping chunk to zero in the overlapping region and joins the bidirectional encoder embedding of the second overlapping chunk with the zero.

17. The computer program according to claim 15, wherein the joining step includes averaging the bidirectional encoder embedding of the first overlapping chunk and the bidirectional encoder embedding of the second overlapping chunk with the bidirectional encoder embedding of the second overlapping chunk in the overlapping regions.

18. A memory device for storing program code, A speech recognition system using a recurrent neural network converter model, comprising a processor device operably coupled to the storage device for executing the program code, wherein the program code is Receiving digital audio and The digital audio is divided into multiple overlapping chunks, wherein two overlapping chunks adjacent to each other have overlapping regions. To obtain a bidirectional encoder embedding, the bidirectional encoder embedding is calculated for each of the multiple overlapping chunks, The aforementioned bidirectional encoder embedding is combined, Using the aforementioned coupled bidirectional encoder embedding, speech recognition of the digital audio is performed, A speech recognition system that performs this task.

Images