Speech recognition method and device based on improved Transformer model

Abstract

The present invention relates to a speech recognition method based on an improved Transformer model. Speech recognition is performed using the improved Transformer model. The improved method is a feature fusion method that uses a splicing function and a convolutional neural network to fuse the high- and low-level features of the decoder, and extracts local feature information. The local detail features extracted by the convolutional neural network are fused with the global features of the Transformer, making the features extracted by the model more robust. At the same time, a short-distance backpropagation path is constructed for each layer of the decoder to alleviate the gradient vanishing problem at the bottom layer of the model; and position coding enhancement is performed. The speech feature embedding vector and position coding of the Transformer model are disassembled to solve the problem of noise caused by the weak correlation between the two. Then, position coding information is added to the self-attention sublayer of each encoder layer of the Transformer model to solve the problem of weakening of position coding information due to multiple superpositions of the network.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech recognition method and device based on an improved Transformer model. Background Art

[0002] Automatic speech recognition is a technology that converts speech sequences into text sequences. Speech recognition technology has been widely used in many fields. With the rapid development of the mobile internet in recent years, people's daily lives are no longer limited to human-to-human communication, and the interaction between people and machines is also becoming increasingly frequent. Currently, electronic devices such as mobile phones, tablets, and computers are beginning to have voice control functions. Compared with using touchpads, mice, keyboards, etc., voice control methods are more convenient and natural. Currently, end-to-end speech recognition models have achieved excellent results in speech recognition tasks. Among them, the Transformer model is one of the most popular methods and is widely used in the field of speech recognition. Although the Transformer has achieved good results in speech recognition, it still has some shortcomings:

[0003] 1. The Transformer model lacks attention to local information. In speech recognition, the individual words in a sentence are often not independent but contextually dependent. The self-attention mechanism used in the Transformer models global features, which can lead to a lack of attention to local features. Therefore, it is necessary to compensate for the Transformer's local feature information. Furthermore, the Transformer only uses features extracted from the top layer for prediction and recognition. If appropriate methods could be used to incorporate detailed information from different layers into the final prediction and recognition, the model could achieve better performance. Furthermore, when the neural network is deep, the underlying gradient updates undergo multiple multiplications, which poses the risk of vanishing gradients.

[0004] 2. After stacking multiple neural networks, the model's positional encoding information is weakened in the semantic space, which can reduce model performance. The Transformer avoids using a recurrent structure and instead relies solely on a self-attention mechanism. Therefore, the model cannot discern positional information in the input sequence. For the Transformer to perceive sequence order, sequence position marker information, known as positional encoding, must be added to the input feature vector. Meanwhile, after passing through multiple neural networks, the semantic features of the input sequence are extracted. However, the positional information contained in the positional encoding is weakened after passing through multiple layers of the Transformer. Furthermore, the Transformer's self-attention layer incorporates both the input sequence and the positional encoding, which introduces unnecessary noise when calculating their associations through dot products.

[0005] As mentioned above, traditional Transformer-based speech recognition models lack attention to local information and ignore some local feature information. Moreover, because the Transformer only uses features extracted from the top layer for prediction and recognition, it ignores detailed information from other layers. The Transformer requires the superposition of multiple layers of neural networks, but deep networks not only cause the underlying gradients to disappear, but also blur the positional encoding. Because the Transformer model is a general speech recognition model and is not specifically optimized for Chinese, homophone errors occurred in previous dataset experiments. Therefore, targeted improvements are needed to meet the requirements of Chinese language recognition. Summary of the Invention

[0006] The purpose of the present invention is to solve at least one of the deficiencies of the prior art and to provide a speech recognition method and device based on an improved Transformer model.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] Specifically, a speech recognition method based on an improved Transformer model is proposed, including the following:

[0009] Obtaining voice data to be recognized;

[0010] Preprocessing the voice data to obtain preprocessed voice data;

[0011] Performing speech recognition on the speech data using an improved Transformer model to obtain a speech recognition result;

[0012] Outputting the speech recognition result;

[0013] The difference between the improved Transformer model and the traditional Transformer model is that

[0014] The feature fusion method is to use the splicing function and convolutional neural network to fuse the high- and low-level features of the decoder, extract local feature information, and fuse the local detail features extracted by the convolutional neural network with the global features of the Transformer; as well as position coding enhancement, the speech feature embedding vector and position coding of the Transformer model are disassembled, and then the position coding information is added to the self-attention sublayer of each encoder layer of the Transformer model.

[0015] Furthermore, specifically, the concatenation function and convolutional neural network are used to fuse the high- and low-level features of the decoder, and local feature information is extracted. The local detail features extracted by the convolutional neural network are fused with the global features of the Transformer, including:

[0016] First, all layer features except the highest layer are concatenated for better fusion. Then, a convolutional neural network is used to compress and extract the concatenated features to ensure that the output after fusion has the same dimension as the features of each decoding layer. The formula is as follows:

[0017]

[0018] Among them, SFF is the algorithm definition name, In the decoder The output, Indicates the batch size (batchsize), represents a single embedding amount, It is a splicing operation. , that is, the concatenation operation concatenates the input matrices along the first dimension, is a compression function. The SFF algorithm uses convolutional neural networks to achieve compression, which makes SFF return to , since the features from each decoder are If the dimensions are different, the extracted fusion features will not be superimposed on the top layer of the original decoder. In this way, the top-level output features It can be superimposed with the features extracted by SFF.

[0019] Furthermore, specifically, the speech feature embedding vector and positional encoding of the Transformer model are disassembled, and then the positional encoding information is added to the self-attention sublayer of each encoder layer of the Transformer model, including:

[0020] The encoder only uses the speech embedding vector as input. After that, the positional encoding is added to each self-attention layer of the encoder. In the self-attention layer of the encoder, the correlation between the speech embedding vector and the positional encoding is performed separately. During the calculation process, the positional encoding uses its own independent self-attention matrix. The speech embedding vector and the positional encoding are respectively correlated and fused together to generate the attention distribution. The calculation formula of the Transformer encoder self-attention layer is as follows:

[0021]

[0022] in, , is the input of the self-attention layer, is the position code, Consistent with the classic Transformer model, they are the mapping matrices of query, key, and value. and These are two mapping matrices added to the sub-attention layer by position encoding enhancement, namely the query and key mapping matrices independent of position encoding.

[0023] The present invention also proposes a speech recognition device based on an improved Transformer model, comprising the following:

[0024] A voice data acquisition module, used to acquire voice data to be recognized;

[0025] A preprocessing module, configured to preprocess the voice data to obtain preprocessed voice data;

[0026] A speech recognition module, configured to perform speech recognition on the speech data using an improved Transformer model to obtain a speech recognition result;

[0027] A result output module, used for outputting the speech recognition result;

[0028] The difference between the improved Transformer model and the traditional Transformer model is that

[0029] The feature fusion method is to use the splicing function and convolutional neural network to fuse the high- and low-level features of the decoder, extract local feature information, and fuse the local detail features extracted by the convolutional neural network with the global features of the Transformer; as well as position coding enhancement, the speech feature embedding vector and position coding of the Transformer model are disassembled, and then the position coding information is added to the self-attention sublayer of each encoder layer of the Transformer model.

[0030] Furthermore, specifically, the concatenation function and convolutional neural network are used to fuse the high- and low-level features of the decoder, and local feature information is extracted. The local detail features extracted by the convolutional neural network are fused with the global features of the Transformer, including:

[0031] First, all layer features except the highest layer are concatenated for better fusion. Then, a convolutional neural network is used to compress and extract the concatenated features to ensure that the output after fusion has the same dimension as the features of each decoding layer. The formula is as follows:

[0032]

[0033] Among them, SFF is the algorithm definition name, In the decoder The output, Indicates the batch size (batchsize), represents a single embedding amount, It is a splicing operation. , that is, the concatenation operation concatenates the input matrices along the first dimension, is a compression function. The SFF algorithm uses convolutional neural networks to achieve compression, which makes SFF return to , since the features from each decoder are If the dimensions are different, the extracted fusion features will not be superimposed on the top layer of the original decoder. In this way, the top-level output features It can be superimposed with the features extracted by SFF.

[0034] Furthermore, specifically, the speech feature embedding vector and positional encoding of the Transformer model are disassembled, and then the positional encoding information is added to the self-attention sublayer of each encoder layer of the Transformer model, including:

[0035] The encoder only uses the speech embedding vector as input. After that, the positional encoding is added to each self-attention layer of the encoder. In the self-attention layer of the encoder, the correlation between the speech embedding vector and the positional encoding is performed separately. During the calculation process, the positional encoding uses its own independent self-attention matrix. The speech embedding vector and the positional encoding are respectively correlated and fused together to generate the attention distribution. The calculation formula of the Transformer encoder self-attention layer is as follows:

[0036]

[0037] in, , is the input of the self-attention layer, is the position code, Consistent with the classic Transformer model, they are the mapping matrices of query, key, and value. and These are two mapping matrices added to the sub-attention layer by position encoding enhancement, namely the query and key mapping matrices independent of position encoding.

[0038] The beneficial effects of the present invention are:

[0039] The speech recognition method based on the improved Transformer model provided by the present invention performs speech recognition using the improved Transformer model. The improvement involves feature fusion, which utilizes a concatenation function and a convolutional neural network to fuse high- and low-level features of the decoder and extract local feature information. The local detail features extracted by the convolutional neural network are then integrated with the global features of the Transformer, making the features extracted by the model more robust. A short backpropagation path is also constructed for each decoder layer to alleviate the vanishing gradient problem at the bottom layer of the model. Furthermore, positional encoding is enhanced by decomposing the speech feature embedding vector and positional encoding of the Transformer model to address the noise caused by the weak correlation between the two. Positional encoding information is then added to the self-attention sublayer of each encoder layer of the Transformer model to address the weakening of positional encoding information due to multiple network stacking. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and other features of the present disclosure will become more apparent through a detailed description of the embodiments shown in conjunction with the accompanying drawings. The same reference numerals in the drawings of the present disclosure represent the same or similar output voltages. Obviously, the drawings described below are only some embodiments of the present disclosure. It is possible for a person skilled in the art to derive other drawings based on these drawings without inventive effort. In the drawings:

[0041] Figure 1 FIG2 is a flow chart of a speech recognition method based on an improved Transformer model of the present invention;

[0042] Figure 2 FIG2 is a schematic diagram showing a feature fusion algorithm of a speech recognition method based on an improved Transformer model according to the present invention;

[0043] Figure 3 FIG2 is a schematic diagram of the structure of a Transformer encoder using a fusion position coding enhancement algorithm in a speech recognition method based on an improved Transformer model of the present invention;

[0044] Figure 4 The figure shows the principle diagram of the position coding enhancement algorithm in the self-attention layer of the encoder of the speech recognition method based on the improved Transformer model of the present invention;

[0045] Figure 5 Shown is the structural principle diagram of the DF method;

[0046] Figure 6 Shown is the structural principle diagram of the LF method;

[0047] Figure 7 The figure shows the overall framework of the traditional Transformer model;

[0048] Figure 8 Shown is the application of positional encoding in a traditional Transformer encoder. DETAILED DESCRIPTION

[0049] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments and features in the embodiments of this application can be combined with each other unless there is a conflict. The same reference numerals used throughout the drawings indicate the same or similar parts.

[0050] Reference Figure 7 First, we introduce the Transformer model in the existing technology.

[0051] The Transformer model is a model architecture that abandons recursion and relies solely on the attention mechanism, allowing for greater parallelism. As an end-to-end model, the Transformer is also based on an encoder-decoder architecture. This requires the encoder to be sufficiently powerful in encoding the input sequence to ensure that the context embedding vector of the input sequence retains as much information as possible about the input sequence. As sentence length increases, the model can no longer fully retain most of the information in the input sequence, and the expressive power of the context embedding vector gradually decreases. Thanks to the attention mechanism, the Transformer no longer considers the encoder's hidden state at the last time step. Instead, at each time step, the decoder automatically finds the hidden state that most closely correlates between the input and output. These hidden states, along with the previous decoder output, are then used as the next input to the decoder. In these end-to-end architectures, the encoder maps the input symbol sequence into a continuous sequence of representations. After obtaining the representation sequence, the decoder generates an output sequence consisting of element-by-element symbols. At each step, the decoder performs autoregression. This means that when performing the next transformation, the decoder uses the symbols generated in the previous step as additional input. Transformer also follows this overall architecture, stacking self-attention layers with feedforward neural network layers in the encoder and decoder. Its overall architecture is as follows Figure 7 As shown, the left and right halves represent the encoder and decoder respectively.

[0052] The Transformer encoder is composed of N identical layers connected together. Each layer contains two sublayers. In the encoder layer, the input sequence first passes through a multi-layer self-attention network sublayer before being input into a feedforward neural network sublayer. A residual connection is used between these two sublayers to input residual information and accelerate backpropagation. The feature sequence after adding the residual is then subjected to layer normalization.

[0053] The decoder also consists of N identical decoder layers. In addition to the self-attention and feedforward neural network sublayers common to the encoder layer, the decoder layer also adds a third sublayer between them, which passes the encoder output through a multi-head self-attention network again. Similar to the encoder, a residual connection is used between every two sublayers, followed by layer normalization.

[0054] Reference Figure 1 The present invention proposes a speech recognition method based on an improved Transformer model, comprising the following steps:

[0055] Step 110: Acquire voice data to be recognized;

[0056] Step 120: preprocessing the voice data to obtain preprocessed voice data;

[0057] Step 130: Perform speech recognition on the speech data using the improved Transformer model to obtain a speech recognition result;

[0058] Step 140: output the speech recognition result;

[0059] The difference between the improved Transformer model and the traditional Transformer model is that

[0060] The feature fusion method is to use the splicing function and convolutional neural network to fuse the high- and low-level features of the decoder, extract local feature information, and fuse the local detail features extracted by the convolutional neural network with the global features of the Transformer; as well as position coding enhancement, the speech feature embedding vector and position coding of the Transformer model are disassembled, and then the position coding information is added to the self-attention sublayer of each encoder layer of the Transformer model.

[0061] Reference Figure 2 As a preferred embodiment of the present invention, specifically, the high-level and low-level features of the decoder are fused using the splicing function and the convolutional neural network, and local feature information is extracted, and the local detail features extracted by the convolutional neural network are fused with the global features of the Transformer, including:

[0062] Speech Feature Fusion Method (SFF) Based on Convolutional Neural Network

[0063] Because the self-attention mechanism in the Transformer models global features, its ability to capture local information in the embedding vector is weaker than that of RNNs and CNNs. In convolutional neural networks, the convolution kernel operates only within the receptive field, which is small, so the convolution operation extracts local features. The SFF algorithm uses a convolutional network in the Transformer decoder, leveraging the local nature of convolutional networks to strengthen the Transformer's focus on local features. Secondly, the Transformer only uses features extracted from the top layer for prediction and recognition. Compared to fusing multi-layer features, the features used for prediction in the previous approach are not sufficiently representative, thus failing to achieve optimal model performance. The SFF algorithm uses feature fusion to fully incorporate features from both high- and low-level layers, making the extracted features more representative and the model more robust. Furthermore, the Transformer suffers from the problem of vanishing gradients at the bottom layer. The Transformer is essentially composed of residual connections and layer normalization modules.

[0064] In summary, the SFF algorithm improves the robustness of the Transformer in speech recognition by addressing the aforementioned issues. The speech feature fusion algorithm applied to the Transformer decoder can be used to fuse features from various layers within the decoder. It also utilizes convolutional neural networks to address the problem of insufficient local features and leverages additional connections between the bottom and top layers to address the potential vanishing gradient problem at the bottom layer. This paper employs an effective approach to feature fusion. First, features from all layers except the top layer are concatenated for better fusion. A convolutional neural network is then used to compress and extract these concatenated features, ensuring that the SFF algorithm's fused output has the same dimensionality as the features at each decoding layer.

[0065] The SFF algorithm combines features from all layers except the top layer, known as high- and low-level features. It then uses a convolutional neural network to extract local features from these layers. Finally, these extracted high- and low-level features are integrated into the top layer of the decoder, making the predicted features more comprehensive.

[0066] Its formula is expressed as follows:

[0067]

[0068] Among them, SFF is the algorithm definition name, In the decoder The output, Indicates the batch size (batchsize), represents a single embedding amount, It is a splicing operation. , that is, the concatenation operation concatenates the input matrices along the first dimension, is a compression function. The SFF algorithm uses convolutional neural networks to achieve compression, which makes SFF return to , since the features from each decoder are If the dimensions are different, the extracted fusion features will not be superimposed on the top layer of the original decoder. In this way, the top-level output features It can be superimposed with the features extracted by SFF.

[0069] The structure of the SFF algorithm is as follows Figure 2 As shown in Figure 2, the SFF algorithm generally fuses features from all decoder layers, resulting in more representative features. Secondly, it uses convolution to extract local information from these fused features, strengthening the model's focus on local features. The model then adds the fused features extracted by the SFF algorithm to the top-level features, fully leveraging features from all layers. Furthermore, by adding features from each layer to the top-level layer, SFF creates a short path from the top-level layer to all other layers, facilitating backpropagation in the neural network model and addressing the vanishing gradient issue at the bottom-level layers.

[0070] In order to verify the feasibility of the SFF algorithm, the high-level and low-level features and local feature methods of the Transformer decoder are studied.

[0071] This paper designs two methods and uses them in the Transformer decoder to further explore the role of high-level and low-level features and local feature extraction in the model. The first method is the high-level and low-level direct fusion method (DF method for short). The structure of this method is as follows Figure 5 As shown in Figure 2, the DF method integrates the first to the second layers in the same way as SFF. First, the DF method transforms the features from layer 1 to layer The decoders are linearly superimposed, and then the layer normalization function is used to smooth the superimposed data. Finally, the integrated high and low layer features are added to the top layer and smoothed again. The structure of the DF method is expressed as

[0072]

[0073] in Indicates the decoder The DF method, like the SFF algorithm, fuses features from each decoder layer. However, the DF method does not use convolution to extract local information. Comparing this method with the SFF algorithm reveals the importance of local information to the model.

[0074] The second method is the single-layer local fusion method, referred to as the LF method. The structure of this method is as follows: Figure 6 As shown in Figure 2. The LF method uses a convolutional neural network in a separate branch to extract local features from the top layer, without using features from other layers. First, LF extracts the top-level feature vector and pads both sides of the vector with blank margins to ensure that the length of the top-level feature vector remains unchanged after convolution. The extracted local features from the top layer are then added to the original top layer. The structure of the LF method is expressed as follows:

[0075]

[0076] in Represented as convolution function, Padding represents edge filling, Indicates the The output of the decoder is the output of the top layer of the decoder. Thus, the DF method and the SFF algorithm both use convolution to extract local information, but they only extract local information from the top layer, discarding information from other layers. Comparing the LF method with the SFF algorithm allows us to study the importance of high- and low-level features to the model.

[0077] Combine Figure 3 as well as Figure 4 As a preferred embodiment of the present invention, specifically, the speech feature embedding vector and position coding of the Transformer model are disassembled, and then the position coding information is added to the self-attention sublayer of each encoder layer of the Transformer model, including:

[0078] The encoder only uses the speech embedding vector as input. After that, the positional encoding is added to each self-attention layer of the encoder. In the self-attention layer of the encoder, the correlation between the speech embedding vector and the positional encoding is performed separately. During the calculation process, the positional encoding uses its own independent self-attention matrix. The speech embedding vector and the positional encoding are respectively correlated and fused together to generate the attention distribution. The calculation formula of the Transformer encoder self-attention layer is as follows:

[0079]

[0080] in, , is the input of the self-attention layer, is the position code, Consistent with the classic Transformer model, they are the mapping matrices of query, key, and value. and These are two mapping matrices added to the sub-attention layer by position encoding enhancement, namely the query and key mapping matrices independent of position encoding.

[0081] Reference Figure 8 ,The feasibility analysis of the position encoding enhancement algorithm (PEE) is as follows,

[0082] The Transformer does not use a recurrent neural network, but instead directly employs an attention mechanism. Without the sequential structure of an RNN, the order information of the input sequences is lost, making it difficult for the model to capture the order of the sequences. To enable the model to perceive the order of the sequences, some information used to mark relative or absolute positions must be injected into the bottom feature vectors of the encoder and decoder, known as positional encoding (PE). To enable the positional encoding to be superimposed on the input sequence, the model sets them to the same vector dimension. The calculation formula of position encoding is as follows:

[0083]

[0084]

[0085] in is the position of the eigenvector, is its dimension. The position encoding of each dimension corresponds to a sine wave or cosine wave. Since for any fixed offset , All used Therefore, this function can make the model easy to pay attention to the relative position information of the feature sequence. The application of position encoding in Transformer encoder is as follows Figure 8 As shown in the figure, the encoder takes the superposition of position encoding and speech embedding vector as input and performs the self-attention mechanism operation together. After passing the encoder composed of encoder layers, the model inputs its output into the decoder.

[0086] However, during training, the model uses multiple self-attention mechanisms and feedforward neural networks. Consequently, numerous linear transformations are performed. This weakens the semantic information contained in the positional encoding after multiple layers are stacked, rendering it ineffective in representing position information. Furthermore, positional encodings, generated by alternating sine and cosine waves, lack semantic information, eliminating the need for multi-layer neural network feature extraction. Therefore, positional encoding information can be added directly to the higher layers of the encoder.

[0087] On the other hand, in the Transformer, the positional encoding and the speech embedding vector are used together as the input of the neural network. After passing through the encoder self-attention layer, the following matrix dot product will be generated:

[0088]

[0089]

[0090] in represents the input of the speech vector, is the position code, 、 、 They are The mapping matrix, Represents the transpose of the matrix. express forward and Perform correlation calculations process, then:

[0091]

[0092]

[0093] Extracting this process separately, according to the matrix operation rules, multiplying two binomials can produce a quartic:

[0094]

[0095] The above equation shows the decomposition process of the positional encoding (position) and the speech embedding vector (speech) after they are superimposed as the input of the neural network and then mapped in the attention layer. It consists of four parts: speech-speech association, speech-position association, position-speech association, and position-position association.

[0096] However, the types of information contained in speech embeddings and positional encodings differ: speech embeddings represent semantic meaning, while positional encodings represent positional information within a sequence. Research by Liu et al. shows that in a Transformer, each input may contain multiple sentences, and due to the randomness of batch processing, a sentence may appear at any position, resulting in a weak correlation between words and positions. In the second and third terms of the above equation, if the speech embedding is used as the query vector (Q) and the positional encoding is used as the vector of queried relevance (K), then their product is meaningless. Therefore, while directly adding positional encodings to speech embeddings introduces positional information, the multiplication operation during the attention mechanism introduces unnecessary noise.

[0097] Based on the above reasons, the present invention uses an effective modification to the current positional encoding in the Transformer, called the Positional Encoding Enhancement algorithm (PEE). PEE is integrated into the self-attention layer of the Transformer encoder. Unlike the classic Transformer, which uses the speech embedding vector after superimposing the positional encoding as input, the encoder only uses the speech embedding vector as input in the positional encoding enhancement algorithm. After that, the positional encoding is added to each self-attention layer of the encoder, and the self-attention layer using the positional encoding enhancement algorithm is called the positional encoding enhanced self-attention layer. As a result, the Transformer encoder architecture is shown in the figure above. In the Transformer encoder, positional encoding is added to each encoder layer, which strengthens the encoder's attention to the positional relationship in the embedding vector. Moreover, the positional encoding is not simply added to each encoder layer, but is added to the self-attention sub-layer in the encoder layer. In the self-attention layer of the encoder, the correlation between the speech embedding vector and the positional encoding is performed separately. During the calculation process, they use different self-attention matrices, that is, the position encoding uses its own independent self-attention matrix. They perform correlation calculations separately and then merge them together to produce the attention distribution.

[0098] The present invention also proposes a speech recognition device based on an improved Transformer model, comprising the following:

[0099] A voice data acquisition module, used to acquire voice data to be recognized;

[0100] A preprocessing module, configured to preprocess the voice data to obtain preprocessed voice data;

[0101] A speech recognition module, configured to perform speech recognition on the speech data using an improved Transformer model to obtain a speech recognition result;

[0102] A result output module, used for outputting the speech recognition result;

[0103] The difference between the improved Transformer model and the traditional Transformer model is that

[0104] The feature fusion method is to use the splicing function and convolutional neural network to fuse the high- and low-level features of the decoder, extract local feature information, and fuse the local detail features extracted by the convolutional neural network with the global features of the Transformer; as well as position coding enhancement, the speech feature embedding vector and position coding of the Transformer model are disassembled, and then the position coding information is added to the self-attention sublayer of each encoder layer of the Transformer model.

[0105] As a preferred embodiment of the present invention, specifically, the high-level and low-level features of the decoder are fused using a splicing function and a convolutional neural network, and local feature information is extracted, and the local detail features extracted by the convolutional neural network are fused with the global features of the Transformer, including:

[0106] First, all layer features except the highest layer are concatenated for better fusion. Then, a convolutional neural network is used to compress and extract the concatenated features to ensure that the output after fusion has the same dimension as the features of each decoding layer. The formula is as follows:

[0107]

[0108] Among them, SFF is the algorithm definition name, In the decoder The output, Indicates the batch size (batchsize), represents a single embedding amount, It is a splicing operation. , that is, the concatenation operation concatenates the input matrices along the first dimension, is a compression function. The SFF algorithm uses convolutional neural networks to achieve compression, which makes SFF return to , since the features from each decoder are If the dimensions are different, the extracted fusion features will not be superimposed on the top layer of the original decoder. In this way, the top-level output features It can be superimposed with the features extracted by SFF.

[0109] As a preferred embodiment of the present invention, specifically, the speech feature embedding vector and position coding of the Transformer model are disassembled, and then the position coding information is added to the self-attention sublayer of each encoder layer of the Transformer model, including:

[0110] The encoder only uses the speech embedding vector as input. After that, the positional encoding is added to each self-attention layer of the encoder. In the self-attention layer of the encoder, the correlation between the speech embedding vector and the positional encoding is performed separately. During the calculation process, the positional encoding uses its own independent self-attention matrix. The speech embedding vector and the positional encoding are respectively correlated and fused together to generate the attention distribution. The calculation formula of the Transformer encoder self-attention layer is as follows:

[0111]

[0112] in, , is the input of the self-attention layer, is the position code, Consistent with the classic Transformer model, they are the mapping matrices of query, key, and value. and These are two mapping matrices added to the sub-attention layer by position encoding enhancement, namely the query and key mapping matrices independent of position encoding.

[0113] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

[0114] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules.

[0115] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0116] Although the present invention has been described in considerable detail and with particularity with respect to several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but rather should be construed as providing a broad possible interpretation of these claims in view of the prior art by reference to the appended claims, thereby effectively encompassing the intended scope of the invention. In addition, the invention has been described above in terms of embodiments foreseen by the inventors for the purpose of providing a useful description, and those insubstantial modifications of the invention that are not currently foreseen may still represent equivalent modifications of the invention.

[0117] The above description is merely a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. As long as the technical effects of the present invention are achieved by the same means, they shall fall within the scope of protection of the present invention. Within the scope of protection of the present invention, various modifications and variations of the technical solutions and / or implementation methods may be made.

Claims

1. A speech recognition method based on an improved Transformer model, characterized in that: These include: Obtaining voice data to be recognized; Preprocessing the voice data to obtain preprocessed voice data; Performing speech recognition on the speech data using an improved Transformer model to obtain a speech recognition result; Outputting the speech recognition result; The difference between the improved Transformer model and the traditional Transformer model is that The feature fusion method uses a concatenation function and a convolutional neural network to fuse the high- and low-level features of the decoder, extract local feature information, and fuse the local detail features extracted by the convolutional neural network with the global features of the Transformer. Furthermore, positional encoding enhancement is performed by decomposing the speech feature embedding vector and positional encoding of the Transformer model, and then incorporating the positional encoding information into the self-attention sublayer of each encoder layer of the Transformer model. The concatenation function and convolutional neural network are used to fuse the high- and low-level features of the decoder and extract local feature information. The local detail features extracted by the convolutional neural network are then integrated with the global features of the Transformer, including: First, all layer features except the highest layer are concatenated for better fusion. Then, a convolutional neural network is used to compress and extract the concatenated features to ensure that the output after fusion has the same dimension as the features of each decoding layer. The formula is as follows: Among them, SFF is the algorithm definition name, In the decoder The output, Indicates the batch size (batchsize), represents a single embedding amount, It is a splicing operation. , that is, the concatenation operation concatenates the input matrices along the first dimension, is a compression function. The SFF algorithm uses convolutional neural networks to achieve compression, which makes SFF return to , since the features from each decoder are If the dimensions are different, the extracted fusion features will not be superimposed on the top layer of the original decoder. In this way, the top-level output features It can be superimposed with the features extracted by SFF.

2. The speech recognition method based on the improved Transformer model according to claim 1, characterized in that: Specifically, the speech feature embedding vector and positional encoding of the Transformer model are disassembled, and then the positional encoding information is added to the self-attention sublayer of each encoder layer of the Transformer model, including: The encoder only uses the speech embedding vector as input. After that, the positional encoding is added to each self-attention layer of the encoder. In the self-attention layer of the encoder, the correlation between the speech embedding vector and the positional encoding is performed separately. During the calculation process, the positional encoding uses its own independent self-attention matrix. The speech embedding vector and the positional encoding are respectively correlated and fused together to generate the attention distribution. The calculation formula of the Transformer encoder self-attention layer is as follows: in, , is the input of the self-attention layer, is the position code, Consistent with the classic Transformer model, they are the mapping matrices of query, key, and value. and These are two mapping matrices added to the sub-attention layer by position encoding enhancement, namely the query and key mapping matrices independent of position encoding.

3. A speech recognition device based on an improved Transformer model, characterized in that: These include: A voice data acquisition module, used to acquire voice data to be recognized; A preprocessing module, configured to preprocess the voice data to obtain preprocessed voice data; A speech recognition module, configured to perform speech recognition on the speech data using an improved Transformer model to obtain a speech recognition result; A result output module, used for outputting the speech recognition result; The difference between the improved Transformer model and the traditional Transformer model is that The feature fusion method uses a concatenation function and a convolutional neural network to fuse the high- and low-level features of the decoder, extract local feature information, and fuse the local detail features extracted by the convolutional neural network with the global features of the Transformer. Furthermore, positional encoding enhancement is performed by decomposing the speech feature embedding vector and positional encoding of the Transformer model, and then incorporating the positional encoding information into the self-attention sublayer of each encoder layer of the Transformer model. Specifically, the concatenation function and convolutional neural network are used to fuse the high- and low-level features of the decoder, extract local feature information, and fuse the local detail features extracted by the convolutional neural network with the global features of the Transformer, including: First, all layer features except the highest layer are concatenated for better fusion. Then, a convolutional neural network is used to compress and extract the concatenated features to ensure that the output of the fused features has the same dimension as the features of each decoding layer. The formula is as follows: Among them, SFF is the algorithm definition name, In the decoder The output, Indicates the batch size (batchsize), represents a single embedding amount, It is a splicing operation. , that is, the concatenation operation concatenates the input matrices along the first dimension, is a compression function. The SFF algorithm uses convolutional neural networks to achieve compression, which makes SFF return to , since the features from each decoder are If the dimensions are different, the extracted fusion features will not be superimposed on the top layer of the original decoder. In this way, the top-level output features It can be superimposed with the features extracted by SFF.

4. The speech recognition device based on the improved Transformer model according to claim 3, characterized in that: The speech feature embedding vector and positional encoding of the Transformer model are disassembled, and then the positional encoding information is added to the self-attention sublayer of each encoder layer of the Transformer model, including: The encoder only uses the speech embedding vector as input. After that, the positional encoding is added to each self-attention layer of the encoder. In the self-attention layer of the encoder, the correlation between the speech embedding vector and the positional encoding is performed separately. During the calculation process, the positional encoding uses its own independent self-attention matrix. The speech embedding vector and the positional encoding are respectively correlated and fused together to generate the attention distribution. The calculation formula of the Transformer encoder self-attention layer is as follows: in, , is the input of the self-attention layer, is the position code, Consistent with the classic Transformer model, they are the mapping matrices of query, key, and value. and These are two mapping matrices added to the sub-attention layer by position encoding enhancement, namely the query and key mapping matrices independent of position encoding.

Images