Low-computing-resource multi-level architecture streaming speech recognition method

Abstract

The invention provides a low-computing-resource multi-level architecture streaming speech recognition method, which is characterized in that an optimized RNN-T streaming speech recognition architecture and a CTC streaming speech recognition architecture are combined, a multi-level mixed streaming speech recognition architecture is proposed, different levels of encoders under the RNN-T architecture are multiplexed as CTC speech recognition encoders, and a multi-level mixed streaming speech recognition architecture is provided. The complexity of the streaming speech recognition model is integrally reduced, the resource occupation flexibility of the model in end-side reasoning is improved, the architecture supports adaptive selection of modules with different complexity for the equipment CPU state at the edge equipment reasoning moment to carry out streaming reasoning, the recognition function of the speech recognition model under the condition of extremely low computing resources is ensured, and the speech recognition efficiency is improved. The voice recognition function can still be completed in the sudden computing resource shortage environment of the end-side equipment, and the safety of a voice recognition system and the stability of the end-side equipment system are guaranteed.

Description

Technical Field

[0001] The present application relates to the field of end-side speech recognition, and is an application method of a speech recognition system architecture solution with low computing resource consumption and a streaming speech recognition technology with adaptive end-side computing resources. Background Art

[0002] The development of deep learning technology has driven the progress of speech recognition technology. At present, server-side speech recognition technology has made great breakthroughs and has surpassed humans in terms of recognition accuracy. However, due to the limitation of computing resources on the client side, the client-side speech recognition technology cannot show the same powerful recognition performance as server-side speech recognition technology. In the more extreme case of low computing resources, the speech recognition function cannot even be enabled normally. Streaming speech recognition refers to continuously feeding speech information into the speech recognition model, and then streaming the speech recognition text results to achieve the effect of outputting the text while the speaker is speaking. When the model processes a frame of speech less than the duration of a frame of speech itself, it can achieve real-time effect and the delay of speech recognition can be basically ignored. However, due to the complexity of the speech recognition model, whether the model supports the streaming architecture, and the influence of the computing resources available on the client side, many current mainstream speech recognition models do not support the requirements of low computing resource client-side streaming speech recognition. When the computing power of the client-side device is limited, the speech recognition function cannot be completed or the recognition delay that users cannot accept occurs. The current mainstream streaming speech recognition technology architecture is the RNN-T architecture, which consists of three parts: encoder, decoder and connector. The encoder is responsible for processing audio acoustic related information, the decoder is responsible for processing linguistic related information, and the connector combines the encoder and decoder output information as input and finally calculates the text prediction.

[0003] The CTC architecture is also one of the current mainstream streaming speech recognition architectures. It only consists of an encoder, which directly processes the audio into text prediction probabilities. Compared with the RNN-T architecture, it has lower complexity. However, the CTC architecture lacks contextual text information input, so the recognition accuracy will be reduced. Summary of the invention

[0004] The present invention aims to solve the problems of high latency and unstable recognition of streaming speech recognition function on the terminal device, so that the terminal device can still meet the requirements of speech recognition function under extreme conditions. The present invention proposes an optimized CTC, RNN-T hybrid speech recognition architecture speech recognition system, which realizes a system solution for adaptively selecting the speech recognition inference structure according to the remaining computing resources of the terminal device, thereby ensuring the stable operation of the speech recognition model on the terminal device.

[0005] The present invention provides a low-computing-resource streaming speech recognition system solution, comprising:

[0006] S1: CTC, RNN-T hybrid architecture design, setting CTC loss in the middle depth layer of the encoder in the RNN-T architecture, and setting CTC loss at different output layer positions of the encoder, so that the RNN-T structure encoder can be divided into a multi-level CTC encoder structure, and different encoder modules can be selected for recognition and reasoning according to the actual remaining computing resources.

[0007] S2: Dynamic inference architecture. When the client-side speech recognition model is running, the CPU usage is monitored in real time and the inference scheme is dynamically selected based on the remaining CPU usage. When the CPU usage is small, the first-level CTC inference scheme is applied. When the CPU usage is medium, the second-level CTC inference scheme is applied. When the CPU usage is sufficient, the RNN-T inference scheme is applied.

[0008] The beneficial effects of the present invention are: low computational complexity, suitable for low computing resource platforms; high stability of the recognition system, when the system CPU is unstable, it can still stably complete the speech recognition function without causing excessive delay; the system has high adaptability, and multi-level recognition schemes take into account both recognition accuracy and delay function, and more combinations can adapt to different scenario requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 This is the CTC and RNN-T hybrid architecture diagram of the present invention

[0010] Figure 2 It is the dynamic reasoning flow chart of the present invention.

[0011] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings.

[0012] Specific implementation measures

[0013] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0014] S1: The input audio data is first framed and windowed, the frame length is set to 25ms, the frame shift is set to 10ms, and each frame is Fourier transformed to extract 80-dimensional Fbank features as the input features of the audio. The input audio is encoded using the zipformer encoder structure. The input first passes through the embedding layer to extract the embedded features and performs 2x time domain downsampling. Then, it is serially connected through the downsampling encoding blocks with downsampling factors of 2, 4, 8, and 2. After passing through a block, it will be upsampled and transformed back to the original time domain resolution. Finally, the encoder output is obtained through the output layer with a downsampling factor of 2. For example, the input audio feature is 100Hz, and the embedded output of 50Hz is obtained after the embedding layer. Then, it passes through the encoding block in series to encode 25Hz, 12.5Hz, 6.25Hz, and 25Hz respectively, and finally, the final encoded output of 25Hz is obtained through the output downsampling layer. The following formula is the overall encoding process of the encoder module:

[0015] H=D2(U(EB2(U(EB8(U(EB4(U(EB2(E(X)))))))))))

[0016] Where E(X) represents the embedding layer, EB i Indicates the encoding block with downsampling factor i, and U indicates recovery to the EB i The original sampling rate is used, D2 represents 2 times downsampling, and H is the acoustic output finally obtained by the encoder.

[0017] S2: Introduce the RNN-T architecture.

[0018] One-dimensional convolution is used as the decoder structure, and the connector structure is composed of linear layers. The above information predicts the text vector of the following text through the decoder structure. The text vector is combined with the acoustic vector output by the encoder and input into the connector structure to obtain the final text prediction output, that is, to realize the RNN-T architecture speech recognition text prediction.

[0019] S3: Introducing a multi-level CTC architecture.

[0020] like Figure 1, in the block where the encoder downsampling factor is 4, that is, the first level CTC structure is introduced after the second encoder block. First, a linear layer is constructed to project the output of the second encoder block to the text space dimension. The linear layer output is transformed by softmax as the first level CTC output. The encoder output layer introduces an additional linear layer transformation, projects the feature dimension to the word space dimension, and uses it as the second level CTC output after softmax transformation. The introduction of two levels of CTC output can enable the model encoder to have independent CTC output capabilities. In practical applications, the overall recognition accuracy of the model may decrease slightly, but the computational complexity of the CTC1 and CTC2 models is significantly reduced compared to the RNN-T model. Specifically, the computational complexity of the CTC1 model is only 40% of that of the RNN-T, while the computational complexity of the CTC2 model is 80% of that of the RNN-T. This multi-level architecture greatly reduces the overall recognition delay and CPU usage, and is particularly suitable for scenarios with low computing resources for streaming speech recognition. The above multi-level CTC architecture is shown in the following formula:

[0021] H2=U(EB4(U(EB2(E(X)))))

[0022] CTC1=Softmax(Linear T (H2)

[0023] CTC2=Softmax(Linear W (E output ))

[0024] S4: Hybrid architecture training mode.

[0025] The model training scheme uses the joint training method of CTC and RNN-T. The loss function consists of three parts. The first part is the first-level CTC loss, the second part is the second-level CTC loss, and the third part is the RNN-T loss. The final function loss is obtained by the weighted sum of the three. Finally, the summed loss is back-propagated to train the hybrid architecture model.

[0026] S5: End-side reasoning mode.

[0027] After completing the model training, Figure 2 In the client-side inference mode, the model monitors the CPU usage in real time when performing speech recognition inference, and dynamically adjusts the inference method according to different CPU occupancy rates to optimize performance and resource utilization:

[0028] When the CPU usage is higher than 90%, in order to reduce the system burden, the system will switch to the first-level CTC inference mode. This mode is relatively simple and suitable for fast response under high load conditions.

[0029] When the CPU usage is between 70% and 90%, the system uses the second-level CTC inference mode. Compared with the first-level CTC, this mode encoder structure is more complex and can provide higher recognition accuracy while still maintaining good efficiency.

[0030] When the CPU usage is below 70%, the system switches to RNN-T inference mode. Although this mode requires more computation, it provides the best recognition performance and accuracy and is suitable for use in low-load situations.

[0031] The following formula describes the reasoning mode of the hybrid architecture:

[0032]

[0033] Among them, Mode1 is the first-level CTC model, Model2 is the second-level CTC model, and Model3 is the third-level RNN-T model.

[0034] Through such a dynamic adjustment mechanism, the system can flexibly select the most appropriate inference mode under different load conditions, ensuring that resource utilization and response speed are optimized to the maximum extent while meeting performance requirements.

[0035] The present invention has been successfully deployed in multiple low-resource platforms such as Raspberry Pi 4B, and can realize the streaming real-time speech recognition function of the hybrid architecture. It has stable speech recognition performance and risk resistance, and can still complete the speech recognition function under extremely low computing resource conditions.

[0036] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "includes one..." does not exclude the existence of other identical elements in the process, method, article or system including the element. The serial numbers of the above-mentioned embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order, and these words may be interpreted as identifiers.

[0037] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A low-computing-resource multi-level architecture streaming speech recognition method, characterized in that: The following steps are involved: Combining CTC and RNN-T speech recognition architecture, an optimized multi-level streaming speech recognition model architecture is proposed; At the time of inference, the multi-level streaming model dynamically uses different modules of the model for inference based on CPU resource information.

2. The method according to claim 1, characterized in that: RNN-T and multi-level CTC reuse encoder parameters. The model has only one set of operating parameters, but different modules in the same set of parameters can be selected for reasoning according to different reasoning modes.

3. The method according to claim 1, characterized in that When recognizing text, different recognition architectures will be adaptively selected based on the current CPU resource situation. When the device CPU is extremely highly occupied, the model selects the lowest-resource CTC inference module for speech recognition. When the device CPU is highly occupied and low computing resources are remaining, the model selects the low-resource CTC inference module for speech recognition inference. When the CPU is low occupied and high computing resources are remaining, the model selects the high-resource RNN-T inference module for inference.

4. The method according to claim 1, characterized in that: Under the high and low CPU resource recognition architecture, the historical text recognized is architecture-independent public information, which can be used as the historical text information of streaming speech recognition. That is, the inference module can be dynamically switched during the streaming recognition process to share the previous text history information.

Images