Converter-based batch voice simultaneous decoding method
Through the converter-based batch voice simultaneous decoding method, the problem of single voice signal input decoding in the prior art is solved, efficient decoding of multiple concurrent inputs is realized, and the computing resource utilization rate and concurrency capability of speech recognition services is improved. Supports the recognition of high concurrency scenarios.
Patent Information
- Application Number
- CN202411030345.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-26
AI Technical Summary
The existing converter-based voice recognition decoder can only perform single-channel voice signal input decoding, resulting in large delay and low computing resource utilization in high concurrent multi-input scenarios, resulting in wasted computing resources.
Using a converter-based batch speech simultaneous decoding method, by inputting multiple voice signals and encoding, initializing the decoding state and prediction results, the maximum output probability is used to select the prediction vector for decoding, supporting the simultaneous decoding of multiple concurrent inputs, and optimizing Redundant neural network operation.
It significantly improves the efficiency of computing resources and the concurrent service capabilities of voice recognition services, reduces redundant computing, and improves the throughput and external output capabilities of end-to-end voice recognition engines.
Smart Images

Figure CN119028363A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of artificial intelligence deep learning speech recognition, and in particular to a transformer-based batch speech simultaneous decoding method. Background Art
[0002] End-to-end speech recognition based on transformers is one of the main mainstream speech recognition technologies. The current decoding technology mainly decodes single speech input. The disadvantages of this single speech decoding are that, first, for high-concurrency multi-input scenarios, the latency of the speech recognition service will be relatively large. At the same time, the utilization rate of computing resources such as GPU for single speech decoding will not be high enough, resulting in a waste of computing resources. Summary of the invention
[0003] 1. Technical issues to be resolved
[0004] In view of the shortcomings of the prior art, the present invention provides a transformer-based batch speech simultaneous decoding method, which solves the problem that the existing transformer-based speech recognition decoder can only perform single-channel speech recognition signal input decoding.
[0005] (II) Technical solution
[0006] To achieve the above object, the present invention provides the following technical solutions:
[0007] A method for simultaneous batch speech decoding based on a transformer comprises the following steps:
[0008] S1: input speech signal, x1, x2, ..., xB;
[0009] S2: output signal, encs = encoder (x) # encoder output signal after encoder changes;
[0010] S3: Initialize decoding, hyps, state = [[]*B], [decoder.init_state()*B] initializes the decoding result to B empty results;
[0011] S4: Initialize the prediction results, predictions = [BOS*B], the initialization prediction results are B sentence start symbols;
[0012] S5: Initialize the indicator vector, b2done, b2t = [False*B], [0*B] initialize the indicator vector;
[0013] S6: Output prediction vector, predictions = argmax(token-probs) For each output layer, select the prediction with the maximum output probability to form an output prediction vector with a shape of [B].
[0014] Furthermore, the output shape in S2 is a three-dimensional matrix of [B, T, dim]B*T*dim.
[0015] Based on the above, the state of the decoder is initialized in S3 for initial calculation of decoding.
[0016] As a further solution of the present invention, b2done in S5 is a B-dimensional vector, in which the i-th dimension vector is True if the i-th voice input has completed decoding, otherwise it is false; b2t is a B-dimensional vector, in which the i-th dimension records the current decoded frame corresponding to the i-th audio.
[0017] Furthermore, it also includes a controller, a voice receiver and an operating device, the voice receiver is electrically connected to the controller, and the controller is electrically connected to the operating device.
[0018] On the basis of the above, the voice receiver receives external operation voice information and sends the operation signal of the operation voice information to the controller, and the controller controls the operation device to execute the operation signal.
[0019] (III) Beneficial effects
[0020] Compared with the prior art, the present invention provides a method for simultaneous batch speech decoding based on a transformer, which has the following beneficial effects:
[0021] 1. The present invention provides an effective decoding method for multiple voice input signals in end-to-end voice recognition based on a transformer. This method can simultaneously decode and recognize multiple voice input signals, improve the utilization efficiency of computing resources GPU, and improve the concurrent service capability of voice recognition services. It supports simultaneous decoding of multiple concurrent inputs, so that voice recognition services can be used in high-concurrency recognition scenarios.
[0022] 2. The decoding algorithm of the present invention further optimizes redundant neural network operations, allowing us to minimize redundant calculations in the case of multiple concurrent inputs. Our algorithm can significantly improve the throughput of the end-to-end speech recognition engine and enhance the external output capabilities of the speech recognition service. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 The figure is a schematic diagram of the process structure of a converter-based batch speech simultaneous decoding method in the present invention. DETAILED DESCRIPTION
[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0025] Example 1
[0026] Reference Figure 1 , a transformer-based batch speech simultaneous decoding method, comprising the following steps:
[0027] 1. Input speech signal, x1, x2, ..., xB;
[0028] 2. Output signal, encs = encoder (x) # The output signal of the encoder after the encoder transformation, where the output shape is a three-dimensional matrix of [B, T, dim] B*T*dim, where B is the number of input speech signals, T is the length of each speech signal, and dim is the dimension of the output after the encoder transformation;
[0029] 3. Initialize decoding, hyps, state = [[] * B], [decoder.init_state() * B] initializes the decoding result to B empty results, and initializes the state of the decoder for the initial calculation of decoding;
[0030] 4. Initialize the prediction results, predictions = [BOS*B], the initial prediction results are B sentence start symbols;
[0031] 5. Initialize the indicator vector, b2done, b2t = [False*B], [0*B] Initialize the indicator vector, b2done is a B-dimensional vector, where the i-th dimension vector is True if the i-th voice input has been decoded, otherwise it is false, b2t is a B-dimensional vector, where the i-th dimension records the current decoded frame corresponding to the i-th audio;
[0032] When not all b2done are true, we do the following:
[0033] 6. decs, states = decoder (state, predictions) calculates the output of the decoder and the state vector corresponding to the decoder;
[0034] 7. token-probs = joiner(encs[b2t], decs) selects B encoder outputs corresponding to b2t, encs[b2t] is a [B, dim] matrix, and combines it with the decoder output decs to calculate the output layer probability distribution token-probs of the transformer transducer, where token-probs is a [B, V] matrix, V is the number of characters corresponding to the output layer;
[0035] 8. Output prediction vector, predictions = argmax(token-probs) For each output layer, select the prediction with the maximum output probability to form an output prediction vector with shape [B];
[0036] 9. b2t[b]+=1where prediction[b]is 0 For those speech frames whose output prediction is a null character, the timestamp is incremented by 1, because the timestamps corresponding to those frames jump to the next moment;
[0037] 10. b2done[b]=true where b2t[b]>audio-length[b] For those input voices whose timestamp length is greater than the corresponding voice length, set the decoding completion sign bit;
[0038] When some predictions are blank characters and b2done is not all true, do the following:
[0039] 11. token-probs = joiner(encs[b2t], decs) selects B encoder outputs corresponding to b2t, encs[b2t] is a [B, dim] matrix, and combines it with the decoder output decs to calculate the output layer probability distribution token-probs of the transformer transducer, where token-probs is a [B, V] matrix, V is the number of characters corresponding to the output layer;
[0040] 12. more-predictions = argmax(token-probs) For each output layer, select the prediction with the maximum output probability to form an output prediction vector of shape [B];
[0041] 13. Update prediction predictions[b] = more-predictions[b] where prediction[b] outputs a blank character;
[0042] 14. b2t[b]+=1where prediction[b]is still 0 For those speech frames whose output prediction is still a null character, the timestamp is increased by 1, because the timestamps corresponding to those frames jump to the next moment;
[0043] 15. b2done[b]=true where b2t[b]>audio-length[b] For those input voices whose timestamp length is greater than the corresponding voice length, set the decoding completion sign bit;
[0044] 16. hyps.append(predictions) adds the prediction results to the end of the corresponding results, where hps is a prediction result set of size B;
[0045] 17. return hyps returns the final prediction result. Our invention can complete the simultaneous decoding of multiple speech recognition input signals. Our decoding method enables our engine to decode multiple speech recognition input signals at the same time, greatly improving the speech recognition engine's support for concurrent input scenarios.
[0046] The present invention also includes a controller, a voice receiver and an operating device. The voice receiver is electrically connected to the controller, and the controller is electrically connected to the operating device. The voice receiver receives external operating voice information and sends an operating signal of the operating voice information to the controller. The controller controls the operating device to execute the operating signal. At the same time, our decoding algorithm further optimizes redundant neural network operations, so that we can minimize redundant calculations in the case of multiple concurrent inputs. Our algorithm can significantly improve the throughput of the end-to-end voice recognition engine and improve the external output capability of the voice recognition service.
[0047] Example 2
[0048] Reference Figure 1 , a transformer-based batch speech simultaneous decoding method, comprising the following steps:
[0049] 1. Input speech signal, x1, x2, ..., xB;
[0050] 2. Output signal, encs = encoder (x) # The output signal of the encoder after the encoder transformation, where the output shape is a three-dimensional matrix of [B, T, dim] B*T*dim, where B is the number of input speech signals, T is the length of each speech signal, and dim is the dimension of the output after the encoder transformation;
[0051] 3. Initialize decoding, hyps, state = [[] * B], [decoder.init_state() * B] initializes the decoding result to B empty results, and initializes the state of the decoder for the initial calculation of decoding;
[0052] 4. Initialize the prediction results, predictions = [BOS*B], the initial prediction results are B sentence start symbols;
[0053] 5. Initialize the indicator vector, b2done, b2t = [False*B], [0*B] Initialize the indicator vector, b2done is a B-dimensional vector, where the i-th dimension vector is True if the i-th voice input has been decoded, otherwise it is false, b2t is a B-dimensional vector, where the i-th dimension records the current decoded frame corresponding to the i-th audio;
[0054] When not all b2done are true, we do the following:
[0055] 6. decs, states = decoder (state, predictions) calculates the output of the decoder and the state vector corresponding to the decoder;
[0056] 7. token-probs = joiner(encs[b2t], decs) selects B encoder outputs corresponding to b2t, encs[b2t] is a [B, dim] matrix, and combines it with the decoder output decs to calculate the output layer probability distribution token-probs of the transformer transducer, where token-probs is a [B, V] matrix, V is the number of characters corresponding to the output layer;
[0057] 8. Output prediction vector, predictions = argmax(token-probs) For each output layer, select the prediction with the maximum output probability to form an output prediction vector with shape [B];
[0058] 9. b2t[b]+=1where prediction[b]is 0 For those speech frames whose output prediction is a null character, the timestamp is incremented by 1, because the timestamps corresponding to those frames jump to the next moment;
[0059] 10. b2done[b]=true where b2t[b]>audio-length[b] For those input voices whose timestamp length is greater than the corresponding voice length, set the decoding completion sign bit;
[0060] When some predictions are blank characters and b2done is not all true, do the following:
[0061] 11. token-probs = joiner(encs[b2t], decs) selects B encoder outputs corresponding to b2t, encs[b2t] is a [B, dim] matrix, and combines it with the decoder output decs to calculate the output layer probability distribution token-probs of the transformer transducer, where token-probs is a [B, V] matrix, V is the number of characters corresponding to the output layer;
[0062] 12. more-predictions = argmax(token-probs) For each output layer, select the prediction with the maximum output probability to form an output prediction vector of shape [B];
[0063] 13. Update prediction predictions[b] = more-predictions[b] where prediction[b] outputs a blank character;
[0064] 14. b2t[b]+=1where prediction[b]is still 0 For those speech frames whose output prediction is still a null character, the timestamp is increased by 1, because the timestamps corresponding to those frames jump to the next moment;
[0065] 15. b2done[b]=true where b2t[b]>audio-length[b] For those input voices whose timestamp length is greater than the corresponding voice length, set the decoding completion sign bit;
[0066] 16. hyps.append(predictions) adds the prediction results to the end of the corresponding results, where hps is a prediction result set of size B;
[0067] 17. return hyps returns the final prediction result. The present invention provides an effective decoding method for multiple voice input signals in end-to-end voice recognition based on transformer. This method can simultaneously decode and recognize multiple voice input signals, improve the utilization efficiency of computing resources GPU, and improve the concurrent service capability of voice recognition service. It supports simultaneous decoding of multiple concurrent inputs, so that voice recognition service can be used in high-concurrency recognition scenarios.
[0068] The present invention also includes a controller, a voice receiver and an operating device. The voice receiver is electrically connected to the controller, and the controller is electrically connected to the operating device. The voice receiver receives external operating voice information and sends an operating signal of the operating voice information to the controller. The controller controls the operating device to execute the operating signal.
[0069] In the description of this article, it should be noted that relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0070] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for simultaneous batch speech decoding based on a transformer, characterized in that: The following steps are involved: S1: input speech signal, x1, x2, ..., xB; S2: output signal, encs = encoder (x) # encoder output signal after encoder changes; S3: Initialize decoding, hyps, state = [[]*B], [decoder.init_state()*B] initializes the decoding result to B empty results; S4: Initialize the prediction results, predictions = [BOS*B], the initialization prediction results are B sentence start symbols; S5: Initialize the indicator vector, b2done, b2t = [False*B], [0*B] initialize the indicator vector; S6: Output prediction vector, predictions = argmax(token-probs) For each output layer, select the prediction with the maximum output probability to form an output prediction vector with a shape of [B].
2. A method for simultaneous batch speech decoding based on a transformer according to claim 1, characterized in that: The output shape in S2 is a three-dimensional matrix of [B,T,dim]B*T*dim.
3. A method for simultaneous batch speech decoding based on a transformer according to claim 2, characterized in that: In S3, the state of the decoder is initialized at the same time for initial calculation of decoding.
4. The method for simultaneous batch speech decoding based on a transformer according to claim 1, characterized in that: In S5, b2done is a B-dimensional vector, in which the i-th dimension vector is True if the i-th voice input has completed decoding, otherwise it is false; b2t is a B-dimensional vector, in which the i-th dimension records the current decoded frame corresponding to the i-th audio.
5. The method for simultaneous batch speech decoding based on a transformer according to claim 1, characterized in that: It also includes a controller, a voice receiver and an operating device. The voice receiver is electrically connected to the controller, and the controller is electrically connected to the operating device.
6. A method for simultaneous batch speech decoding based on a transformer according to claim 5, characterized in that: The voice receiver receives external operation voice information and sends the operation signal of the operation voice information to the controller, and the controller controls the operation device to execute the operation signal.
