A streaming end-to-end speech recognition model, electronic device and storage medium

Abstract

The application discloses a streaming end-to-end speech recognition model and an electronic device, the model is used for being configured in the electronic device, and specifically includes a decoder and an encoder integrated by a joint network. The decoder includes a VGG (Visual Geometry Group) module, which is used for performing convolution processing on an input audio signal by adopting a streaming convolution method; and the encoder includes an LSTM neural network. Since the scheme adopts the streaming convolution and does not depend on next frame information, the time delay problem of the convolution model can be eliminated, so that the time delay of speech recognition is reduced, and the product experience of a user can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, and more particularly to a streaming end-to-end speech recognition model, an electronic device and a storage medium. BACKGROUND

[0002] Speech signal is one of the most common and commonly used signals in human society, and it is an important way for people to express, communicate and disseminate information. In today's era of information explosion, massive amounts of speech data are being generated in the Internet and telephone channels at all times. In order to more efficiently recognize, classify and search large-scale speech signals, the demand for automatic speech recognition (ASR) has become more urgent.

[0003] Compared with the traditional speech recognition system based on Hidden Markov Model (HMM), the streaming end-to-end speech recognition model integrates the acoustic model, pronunciation dictionary and speech model into a set of neural networks, making the model more concise and easier to directly optimize the parameters. In addition, it has stronger universality and faster real-time performance, reduces the dependence on professional linguistic knowledge, and improves the user experience when interacting with the speech product. During training, the training data does not require alignment information, and the labeled speech can be directly used for training; during inference, there is no need to focus on future speech frames, and the information of the current frame can be directly recognized.

[0004] Due to the universality and real-time performance of the streaming end-to-end model, it is favored by the industry. However, in the traditional speech recognition scheme, the input audio features are usually modeled using fixed-size square convolution, which can cause the modeling granularity to be single and the receptive field to be poor, thereby affecting the recognition performance.

[0005] To solve the problem of poor modeling of the streaming end-to-end model, the commonly used method is to increase the complexity of the model modeling. Among them, increasing the complexity of the encoder of the end-to-end model is the most significant for improving the recognition effect. At present, the most popular method is to use the Transformer including Visual Geometry Group (VGG) as the encoding model, use the Long Short-Term Memory (LSTM) neural network as the decoding model, and integrate them through the Joint Network to form a complete speech recognition model.

[0006] In the training stage, in the encoder, the input audio features are obtained by the VGG module to obtain high-level feature representation, and then the attention score is calculated by the Transformer self-attention (SA) mechanism. The score represents the correlation between the high-level audio features. Here, to ensure the streaming nature of the encoder, the mask mechanism is used to eliminate the right visual field information of the SA mechanism, ignore the future frames, and only compare the correlation between the current frame and the historical frames. Then the encoder output is obtained through the linear layer (DenseLayer) and the Dropout layer. In the decoder, the label text sequence is generated by the embedding layer (EmbeddingLayer) to generate the text features, and the decoder output is obtained by the LSTM. Finally, the joint network integrates the encoder and the decoder output, and inputs it into the transducer loss function. The loss function result is propagated backward to update the parameters of the entire network. Here, the transducer loss function can align the high-level representations of the audio frames and the label sequences through the null character.

[0007] In the decoding stage, the decoder has a natural streaming nature. Its input is initialized as a blank character, and with the calculation of the high-level features of each frame of the encoder and the decoder, a probability vector is obtained. The dimension of the vector includes the number of characters corresponding to the pronunciation dictionary and the null character. When the maximum probability is, the new encoder high-level feature is calculated with the decoder. When the maximum probability is a non-null character, it is taken as the new input of the decoder, the decoder LSTM hidden state is updated, and the probability result is calculated again. Through this way, the cycle is repeated until the audio frame input is completed, so as to realize streaming speech recognition.

[0008] In the prior art, by increasing the network layers and neuron node parameters of the VGG module and the Transformer model, the model expression ability can be improved. However, due to the convolution processing method adopted by the VGG module, the right visual field of one frame is needed during decoding, which leads to the increase of the decoding delay with the increase of the number of convolution layers, thereby increasing the overall delay and seriously affecting the product experience of the user. SUMMARY

[0009] Therefore, the present application provides a streaming end-to-end speech recognition model, an electronic device and a storage medium, which are used to reduce the time delay of speech recognition and improve the product experience of the user.

[0010] In order to achieve the above purpose, the present application provides the following scheme:

[0011] A streaming end-to-end speech recognition model is configured in an electronic device, and the streaming end-to-end speech recognition model comprises a decoder and an encoder integrated by a joint network, wherein:

[0012] The decoder comprises a VGG (Visual Geometry Group) module configured to perform convolution processing on the input audio signal by using a streaming convolution method.

[0013] The encoder comprises an LSTM (Long Short-Term Memory) neural network.

[0014] Optionally, the decoder comprises a plurality of output channels, each of which is modeled based on a convolution window of different sizes.

[0015] Optionally, the convolution window is a circular kernel convolution window.

[0016] Optionally, the size of the convolution window is 3x3, 5x5 or 7x7.

[0017] An electronic device, wherein the electronic device is configured with the streaming end-to-end speech recognition model as described above.

[0018] A storage medium, wherein the storage medium carries the streaming end-to-end speech recognition model as described above, and when the electronic device executes the streaming end-to-end speech recognition model, the electronic device is enabled to implement streaming speech recognition.

[0019] As can be seen from the above technical solutions, the present application discloses a streaming end-to-end speech recognition model and an electronic device, the model is configured to the electronic device, and specifically comprises a decoder and an encoder integrated by a joint network. The decoder comprises a VGG (Visual Geometry Group) module configured to perform convolution processing on the input audio signal by using a streaming convolution method; and the encoder comprises an LSTM neural network. Since the present application uses streaming convolution, it does not rely on the next frame information, thus eliminating the time delay problem of the convolution model, reducing the time delay of speech recognition, and further improving the product experience of users. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0021] Figure 1a A schematic diagram of streaming convolution in the embodiments of the present application;

[0022] Figure 1b Schematic diagram of traditional convolution;

[0023] Figure 2a Schematic diagram of the calculation method of traditional convolution;

[0024] Figure 2b Schematic diagram of a calculation method based on convolution of different scales in an embodiment of the present application;

[0025] Figure 3a Schematic diagram of a convolution window with a square kernel;

[0026] Figure 3b Schematic diagram of a convolution window with a circular kernel according to an embodiment of the present application;

[0027] Figure 4 This is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0029] Example 1

[0030] This embodiment provides a streaming end-to-end speech recognition model. This model is applied to electronic devices, which here refer to computers, servers, or mobile devices with data computing and information processing capabilities. This embodiment model includes an encoder as an input and a decoder as an output, which are integrated through a joint network to form a complete streaming end-to-end speech recognition model.

[0031] The decoder of the present application includes a decoding model built based on an LSTM (Long Short-Term Memory) neural network.

[0032] To eliminate the latency caused by convolution, the decoder in this solution uses streaming convolution. Specifically, the encoder's VGG (Visual Geometry Group) module uses streaming convolution to convolve the input audio signal and uses the resulting data as input to the subsequent decoding model to achieve speech recognition.

[0033] like Figure 1aAs shown, the part covered by the oblique line is the audio feature. Among them, the horizontal axis represents the speech frame, and the vertical axis represents the feature dimension. As can be seen from the convolution calculation formula (1), in order to make the feature size consistent before and after the convolution operation, padding processing needs to be performed on the feature. Here, the white part is the part with a convolution padding value of 0.

[0034]

[0035] Among them, x is the input feature dimension, f is the convolution window size, p is the row and column number of padding filling. S is the distance of convolution window sliding (stride) each time, and the stride is usually 1. In Figure 1b The schematic diagram of the traditional convolution is shown in FIG. 1. Figure 1a The schematic diagram of the streaming convolution in the present application is shown in FIG. 2.

[0036] In the decoding stage, the traditional convolution (thick line convolution window) needs two frames of audio features to perform calculation, as shown in FIG. 3; while the streaming convolution (thick line convolution window) calculation only needs one frame of audio features, that is, the corresponding feature can be generated at the same time as the input of one frame of feature, without relying on the next frame of information, as shown in FIG. 4. Figure 1b Figure 1a Therefore, through the streaming convolution, the time delay problem of the convolution model can be eliminated, and the real-time performance of the end-to-end model can be ensured.

[0037] As can be seen from the above technical solution, the embodiment provides a streaming end-to-end speech recognition model, which is configured in an electronic device and specifically includes a decoder and an encoder integrated by a joint network. The decoder includes a VGG (Visual Geometry Group) module, which is configured to perform convolution processing on the input audio signal by using the streaming convolution method; and the encoder includes an LSTM neural network. Since the streaming convolution is used in the present application, the next frame of information is not relied on, so that the time delay problem of the convolution model can be eliminated, thereby reducing the time delay of speech recognition, and further improving the product experience of the user.

[0038] In another specific embodiment of the present application, in order to solve the problem of single granularity of speech modeling primitives, the present application uses multiple output channels, and each output channel is modeled based on different scales of convolution windows.

[0039] Figure 2a The schematic diagram of the calculation method of the traditional convolution is shown in FIG. 1. The T-frame D-dimensional C-channel audio feature is calculated by the 3x3 streaming convolution, and the H-dimensional high-level representation is obtained. However, since only one scale of convolution is used, the modeling strength is single, so the present application proposes multiple convolution windows of different scales, as shown in FIG. 2. Figure 2b ​The input audio features are calculated with three different scale convolution windows, 3x3, 5x5 and 7x7, and the output channels are C3, C5 and C7 respectively. The 3x3 convolution models a smaller granularity, representing the phoneme-level speech primitive; the 5x5 convolution models a moderate granularity, representing the syllable-level speech primitive; and the 7x7 convolution models a larger granularity, representing the sub-word-level speech primitive.

[0040] Then, the three types of output channels containing different speech primitive information are spliced to obtain H-dimensional features. Here, to meet the streaming nature of convolution and the consistency of T frames D dimensions before and after convolution, 2, 4 and 6 frames of constant 0 information are padded before the first frame of the input audio features, and 1, 2 and 3-dimensional all-0 information is padded around the features. Compared with the former, this method has stronger model expression ability.

[0041] Finally, to ensure that the model has a good receptive field, a convolution window with a circular kernel is used for modeling. Although the square kernel convolution window is very suitable for tensor calculation and is often used as the standard modeling unit of VGG. However, the receptive field in the human visual system is actually circular. Inspired by this, the present application proposes a convolution window with a circular kernel. Compared with the square kernel convolution bed, the computational amount required for training of the two schemes is roughly the same. In order to better represent the implementation process, take 3x3 as an example for illustration, as shown in Figure 3a and Figure 3b .

[0042] Among them, Figure 3a is a schematic diagram of a square kernel convolution window, Figure 3b is a schematic diagram of a circular kernel convolution window. By comparison, the latter, the light blue kernel A will move to the kernel O. Finally, all the kernels except kernel D are on a circular track. In square kernel convolution calculation, each kernel will calculate with the corresponding feature to obtain X.

[0043] X = w 1,1 x 1,1 +w 1,2 x 1,2 +w 1,3 x 1,3 +

[0044] w 2,1 x 2,1 +w 2,2 x 2,2 +w 2,3 x 2,3 + (2)

[0045] w 3,1 x 3,1 +w 3,2 x 3,2 +w3,3 x 3,3

[0046] However, in circular kernel calculations, kernel O must be derived from kernels A, B, C, and D. Assume that D is the center origin and the distance to B and C is 1. Using the Euclidean distance approximation, kernel O is 0.42 from kernel A, 0.99 from kernel D, and 0.76 from kernels B and C. The closer the distances, the higher the weight kernel O has in the calculation. For ease of calculation, the reciprocal of the Euclidean distance is used to represent the weight, which yields the calculated kernel O:

[0047] X O =(2.38w 1,1 x 1,1 +1.32w 1,2 x 1,2 +1.32w 2,1 x 2,1 +1.01w 2,2 x 2,2 ) / 4 (3)

[0048] Substituting the results corresponding to the kernel O formula (3) and the three similar kernels into formula (2) yields:

[0049] X=0.60w 1,1 x 1,1 +1.66w 1,2 x 1,2 +0.60w 1,3 x 1,3 +

[0050] 1.66w 2,1 x 2,1 +2.00w 2,2 x 2,2 +1.66w 2,3 x 2,3 + (4)

[0051] 0.60w 3,1 x 3,1 +1.66w 3,2 x 3,2 +0.60w 3,3 x 3,3

[0052] Equation (4) shows that this circular kernel convolution can concentrate feature information as much as possible in the center, giving the model a more reasonable receptive field and preventing information diffusion. In multi-scale streaming convolution, in addition to the 3x3 convolution window, it also includes 5x5 and 7x7 convolution windows. The calculation method for the circular kernel is similar to that for the 3x3, so it will not be repeated here.

[0053] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0054] While operations are depicted in a particular, sequential order, this should not be understood as requiring or implying that the operations are performed in the order depicted or in sequential order, or that all operations be performed. Some operations can potentially be performed in different orders and / or concurrently.

[0055] It will be appreciated that the steps of the method embodiments of the present disclosure can be performed in different order and / or concurrently with each other. Additionally, the method embodiments can include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this regard.

[0056] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0057] Embodiment Two

[0058] Figure 4 A block diagram of an electronic device for embodiments of the present application.

[0059] Reference Figure 4As shown, it shows a structural schematic diagram of an electronic device suitable for implementing the electronic device in the embodiments of the present disclosure, which is configured with the streaming end-to-end speech recognition model disclosed in Embodiment One. The terminal device in the embodiments of the present disclosure can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Personal Computers), PMPs (Portable Multimedia Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, as well as fixed terminals such as digital TVs, desktop computers, and the like. Figure 4 The electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0060] As shown, Figure 4 The electronic device can include a processing device (such as a central processor, a graphics processor, etc.) 401, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 402 or programs loaded from a storage device 406 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0061] Generally, the following devices can be connected to the I / O interface 405: input devices 406 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 406 including, for example, a magnetic tape, a hard disk, and the like; and communication devices 409. The communication devices 409 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device is shown with various devices, but it should be understood that all the devices shown are not required to be implemented or possessed. More or fewer devices can be alternatively implemented or possessed.

[0062] Embodiment Three

[0063] The present embodiment provides a computer-readable storage medium configured with the streaming end-to-end speech recognition model in Embodiment One. It should be noted that the storage medium can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may, for example, be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above.

[0064] More specific examples of the computer-readable storage media can include but are not limited to the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0065] In the present disclosure, a computer readable storage medium can be any tangible medium that can contain, or store computer readable program codes. In the present disclosure, a computer readable signal medium can include a data signal traveling in baseband or traveling as a component of a carrier wave / communication medium. The program codes embodied on the computer readable storage medium or data signals can be transmitted using any suitable medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0066] Each of the embodiments described in the present specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts among the embodiments can be understood by referring to each other.

[0067] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to include all the preferred embodiments and all the changes and modifications falling within the scope of the embodiments of the present application.

[0068] Finally, it needs to be pointed out that in this document, relational terms such as first and second and the like can only be used to distinguish one entity or action from another entity or action, without necessarily requiring or implying any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by an "comprising" statement serves as a means plus function alternative.

[0069] The above detailed description of the technical solutions provided by the present application has been given, and the principles and implementation manners of the present application are described by applying specific examples in this document. The above description of the examples is only for helping to understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will have changes, and the above description of the content of the specification should not be understood as a limitation of the present application.

Claims

1. A streaming end-to-end speech recognition model, configured for an electronic device, characterized in that: The streaming end-to-end speech recognition model includes a decoder and an encoder integrated by a joint network, wherein: The decoder includes a VGG (Visual Geometry Group) module, which is used to perform convolution processing on the input audio signal using a streaming convolution method; The encoder includes an LSTM (Long Short-Term Memory) neural network; The decoder includes multiple output channels, each of which is modeled based on a convolution window of different scales; the sizes of the convolution windows are 3x3, 5x5, and 7x7; the 3x3 convolution modeling granularity is small, representing factor-level speech primitives; the 5x5 convolution modeling granularity is moderate, representing syllable-level speech primitives; and the 7x7 convolution modeling granularity is large, representing subword-level speech primitives. Among them, the method of using streaming convolution to perform convolution processing on the input audio signal includes: splicing three types of output channels containing different speech primitive information to obtain H-dimensional features; filling 2, 4, and 6 frames of information with a constant of 0 before the first frame of the input audio feature, and filling 1, 2, and 3-dimensional all-0 information around the feature.

2. The streaming end-to-end speech recognition model according to claim 1, wherein The convolution window is a convolution window with a circular kernel.

3. An electronic device, characterized in that: The electronic device is configured with the streaming end-to-end speech recognition model according to any one of claims 1 to 2.

4. A storage medium, characterized in that The storage medium carries the streaming end-to-end speech recognition model according to any one of claims 1 to 2, and when the electronic device executes the streaming end-to-end speech recognition model, the electronic device can implement streaming speech recognition.

Images