Speech recognition method and device based on multi-configuration block context transformer model

Abstract

The present invention provides a speech recognition method and device based on a multi-configuration block context transformer model, belonging to the field of artificial intelligence technology, the method comprising: obtaining a feature sequence of a speech to be recognized; inputting the feature sequence into a trained speech recognition model, using the speech recognition model to perform speech recognition on the speech to be recognized, and obtaining a recognition result output by the speech recognition model; the speech recognition model is a multi-configuration block context transformer MCC-Transformer model, the MCC-Transformer model is constructed on the basis of the transformer model by introducing a speech MCC-Transformer block in an encoder, and the speech MCC-Transformer block is formed by stacking multiple MCC-Transformer blocks. The present invention introduces a speech MCC-Transformer block formed by stacking multiple MCC-Transformer blocks in the encoder, thereby improving the accuracy of speech recognition.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a speech recognition method and device based on a multi-configuration block context converter model. Background Art

[0002] The speech recognition model with the Transformer model as the backbone network has been widely used due to its unique self-attention mechanism and network architecture.

[0003] Due to the quadratic complexity of the classic self-attention mechanism, it is difficult for Transformer to train and model long sequence data, which requires a large amount of graphics card memory and prolongs the training time. Chunking is a feasible method to reduce the complexity of the self-attention mechanism. The sequence data is evenly divided into chunks, and the attention between tokens in the chunks is calculated in sequence, which reduces the computational complexity of the self-attention mechanism to linear.

[0004] However, the accuracy of the speech recognition model based on the block mechanism in the existing technology is not high, and the model performance needs to be further improved. Summary of the invention

[0005] The present invention provides a speech recognition method and device based on a multi-configuration block context converter model, which are used to solve the technical problem of low accuracy of speech recognition results in the prior art.

[0006] In a first aspect, the present invention provides a speech recognition method based on a multi-configuration block context transformer model, comprising:

[0007] Obtaining a feature sequence of speech to be recognized;

[0008] Inputting the feature sequence into a trained speech recognition model, performing speech recognition on the speech to be recognized using the speech recognition model, and obtaining a recognition result output by the speech recognition model;

[0009] The speech recognition model is a multi-configuration block context transformer MCC-Transformer model. The MCC-Transformer model is constructed by introducing a speech MCC-Transformer block in the encoder on the basis of the transformer model. The speech MCC-Transformer block is stacked by multiple MCC-Transformer blocks, and each MCC-Transformer block is cascaded in sequence by a multi-head self-attention C-MHSA module based on a uniform blocking mechanism, a multi-head self-attention SC-MHSA module based on a shift blocking mechanism, and a multi-head self-attention SSC-MHSA module based on a sequential sampling blocking mechanism.

[0010] In some embodiments, the C-MHSA module includes a C-MHSA layer, the C-MHSA layer being used to perform attention calculation within a block using uniform partitioning;

[0011] The SC-MHSA module includes a SC-MHSA layer, and the SC-MHSA layer is used to perform attention calculation within the block divided by the shift block;

[0012] The SSC-MHSA module includes an SSC-MHSA layer, which is used to perform attention calculations within blocks that are repartitioned using sequential sampling.

[0013] In some embodiments, the decoder of the MCC-Transformer model includes a CTC decoder and a self-attention decoder;

[0014] The self-attention decoder includes an input layer, an embedding layer, a text MCC-Transformer block, and a speech-text cross MCC-Transformer block;

[0015] The input layer is used to obtain the previous output result of the decoder;

[0016] The Embedding layer is used to contextualize the previous output result of the decoder;

[0017] The text MCC-Transformer block is used to perform a mask operation on the output information of the Embedding layer;

[0018] The speech-text cross MCC-Transformer block is used to calculate a feature matrix based on the output information of the encoder and the output information of the text MCC-Transformer block.

[0019] In some embodiments, the text MCC-Transformer block is composed of multiple MCC-Transformer blocks stacked together, and each MCC-Transformer block is composed of a multi-head self-attention C-MHSA module based on a uniform blocking mechanism, a multi-head self-attention SC-MHSA module based on a shift blocking mechanism, and a multi-head self-attention SSC-MHSA module based on a sequential sampling blocking mechanism.

[0020] In some embodiments, the speech-text cross-MCC-Transformer block is composed of multiple MCC-Transformer blocks stacked together, and each MCC-Transformer block is composed of a multi-head self-attention C-MHSA module based on a uniform blocking mechanism, a multi-head self-attention SC-MHSA module based on a shift blocking mechanism, and a multi-head self-attention SSC-MHSA module based on a sequential sampling blocking mechanism.

[0021] In some embodiments, the encoder of the MCC-Transformer model further includes an input layer, a padding layer, a convolutional layer, a linear mapping layer, and a dropout layer;

[0022] The input layer is used to obtain the feature sequence;

[0023] The padding layer is used to fill zeros in the feature sequence;

[0024] The convolution layer is used to transform the dimension of the feature sequence;

[0025] The linear mapping layer is used to remap the feature sequence;

[0026] The dropout layer is used to set the outputs of some neurons to zero.

[0027] In a second aspect, the present invention further provides a speech recognition device based on a multi-configuration block context transformer model, comprising:

[0028] An acquisition module, used to acquire a feature sequence of the speech to be recognized;

[0029] A recognition module, used to input the feature sequence into a trained speech recognition model, perform speech recognition on the speech to be recognized using the speech recognition model, and obtain a recognition result output by the speech recognition model;

[0030] The speech recognition model is a multi-configuration block context transformer MCC-Transformer model. The MCC-Transformer model is constructed by introducing a speech MCC-Transformer block in the encoder on the basis of the transformer model. The speech MCC-Transformer block is stacked by multiple MCC-Transformer blocks, and each MCC-Transformer block is cascaded in sequence by a multi-head self-attention C-MHSA module based on a uniform blocking mechanism, a multi-head self-attention SC-MHSA module based on a shift blocking mechanism, and a multi-head self-attention SSC-MHSA module based on a sequential sampling blocking mechanism.

[0031] In a third aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a speech recognition method based on a multi-configuration block context converter model as described above is implemented.

[0032] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a speech recognition method based on a multi-configuration block context converter model as described in any one of the above.

[0033] In a fifth aspect, the present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements a speech recognition method based on a multi-configuration block context converter model as described in any one of the above.

[0034] The speech recognition method and device provided by the present invention introduce a speech MCC-Transformer block composed of multiple MCC-Transformer blocks stacked in an encoder, each MCC-Transformer block is sequentially cascaded by a multi-head self-attention C-MHSA module based on a uniform blocking mechanism, a multi-head self-attention SC-MHSA module based on a shift blocking mechanism, and a multi-head self-attention SSC-MHSA module based on a sequential sampling blocking mechanism, thereby improving the accuracy of speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 It is a flow chart of a speech recognition method based on a multi-configuration block context converter model provided by the present invention.

[0037] Figure 2 It is a schematic diagram of the framework of the speech recognition model based on the multi-configuration block context converter model provided by the present invention.

[0038] Figure 3 It is a structural schematic diagram of the MCC-Transformer block provided by the present invention.

[0039] Figure 4 It is a structural schematic diagram of a speech recognition device based on a multi-configuration block context converter model provided by the present invention.

[0040] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] Figure 1 is a flow chart of a speech recognition method based on a multi-configuration block context transformer model provided by the present invention, such as Figure 1 As shown, the method includes the following:

[0043] Step 101: Obtain a feature sequence of speech to be recognized.

[0044] Step 102: input the feature sequence into a trained speech recognition model, use the speech recognition model to perform speech recognition on the speech to be recognized, and obtain a recognition result output by the speech recognition model; the speech recognition model is a multi-configuration block context converter MCC-Transformer model, and the MCC-Transformer model is constructed on the basis of the converter model by introducing a speech MCC-Transformer block in the encoder, and the speech MCC-Transformer block is composed of multiple MCC-Transformer blocks stacked together, and each MCC-Transformer block is cascaded in sequence by a multi-head self-attention C-MHSA module based on a uniform blocking mechanism, a multi-head self-attention SC-MHSA module based on a shift blocking mechanism, and a multi-head self-attention SSC-MHSA module based on a sequential sampling blocking mechanism.

[0045] Specifically, Figure 2 is a schematic diagram of the framework of the speech recognition model based on the multi-configuration block context converter model provided by the present invention, such as Figure 2As shown, the speech recognition model provided by the present invention is a Multi-configuration Chunk-wise Context Transformer (MCC-Transformer) model, which is constructed by introducing a speech MCC-Transformer block in the encoder on the basis of the (traditional) Transformer model. The speech MCC-Transformer block is stacked by multiple MCC-Transformer blocks.

[0046] Figure 3 is a schematic diagram of the structure of the MCC-Transformer block provided by the present invention, such as Figure 3 As shown in the figure, each MCC-Transformer block is composed of a cascade of a Chunk Multi-Head Self-Attention (C-MHSA) module based on a uniform chunking mechanism, a Shifted Chunk Multi-Head Self-Attention (SC-MHSA) module based on a shifted chunking mechanism, and a Sequentially Sampled Chunk Multi-Head Self-Attention (SSC-MHSA) module based on a sequentially sampled chunking mechanism.

[0047] like Figure 3 As shown in the figure, the C-MHSA layer of the C-MHSA module is preceded by a layer normalization (LN) layer, followed by an LN layer, and finally connected to a feed forward network (FFN). In addition, there can be a residual connection (Residual) between the two layers as an auxiliary input.

[0048] The C-MHSA layer is used to perform attention computation within a uniformly divided block.

[0049] The SC-MHSA module has an LN layer in front of the SC-MHSA layer, an LN layer behind it, and finally an FFN. In addition, there can be a residual connection (Residual) between the two layers as an auxiliary input.

[0050] The SC-MHSA layer is used to perform attention calculation within the block partitioned by shifted blocks. The SC-MHSA layer is used to capture the context between neighboring blocks.

[0051] The SSC-MHSA module has an LN layer in front of the SSC-MHSA layer, an LN layer behind it, and finally an FFN. In addition, there can be a residual connection between the two layers as an auxiliary input.

[0052] The SSC-MHSA layer is used to perform attention calculation within the block with sequential sampling repartitioning. The SSC-MHSA layer is used to capture long-range context.

[0053] like Figure 2 As shown, in one embodiment, the speech MCC-Transformer block is composed of 4 stacked MCC-Transformer blocks, with a total of 12 self-attention layers.

[0054] The overall calculation process of the MCC-Transformer block is expressed as follows:

[0055] (1)

[0056] (2)

[0057] (3)

[0058] (4)

[0059] (5)

[0060] (6)

[0061] in, represents the feature vector output by the C-MHSA layer in the C-MHSA module, represents the uniform block self-attention function, represents the regularization function, represents the feature vector input to the C-MHSA module, represents the feature vector output by the C-MHSA module, represents the forward feedback function, represents the feature vector output by the SC-MHSA layer in the SC-MHSA module, represents the shifted block self-attention function, represents the feature vector output by the SC-MHSA module, represents the feature vector output by the SSC-MHSA layer in the SSC-MHSA module, represents the sequential sampling block self-attention function, The feature vector representing the output of the SSC-MHSA module.

[0062] The MCC-Transformer model provided in this application adopts a multi-configuration context self-attention mechanism, which includes three types of contexts. The first context is a local context, that is, the self-attention mechanism calculation is only performed in this block. The second context is a shift block context, that is, the context of the adjacent block is called in through a shift block operation, and the self-attention mechanism is performed in each block after the shift. The third context is a sequential sampling context, that is, the sequence is globally sampled into each block through a sequential sampling operation, and the self-attention mechanism is performed in each block after sequential sampling.

[0063] like Figure 2 As shown, the decoder of the MCC-Transformer model includes a Connectionist Temporal Classification (CTC) decoder and a self-attention decoder;

[0064] The self-attention decoder includes an input layer (not shown in the figure), an embedding layer, a text MCC-Transformer block, and a speech-text cross MCC-Transformer block;

[0065] The input layer is used to obtain the previous output result of the decoder;

[0066] The Embedding layer is used to contextualize the previous output result of the decoder;

[0067] The text MCC-Transformer block is used to perform a mask operation on the output information of the Embedding layer;

[0068] The speech-text cross MCC-Transformer block is used to calculate a feature matrix based on the output information of the encoder and the output information of the text MCC-Transformer block.

[0069] Specifically, in an embodiment of the present application, in the decoder, a hybrid decoding architecture of a CTC decoder and a self-attention decoder is adopted.

[0070] In some embodiments, the text MCC-Transformer block is composed of multiple MCC-Transformer blocks stacked together, and each MCC-Transformer block is composed of a multi-head self-attention C-MHSA module based on a uniform blocking mechanism, a multi-head self-attention SC-MHSA module based on a shift blocking mechanism, and a multi-head self-attention SSC-MHSA module based on a sequential sampling blocking mechanism.

[0071] The structures of the C-MHSA module, SC-MHSA module, and SSC-MHSA module are the same as those used in the encoder and will not be repeated here.

[0072] Specifically, since the length of the text sequence is usually short, in order to ensure that the speech-text cross-MCC-Transformer block can smoothly execute the cross-modal chunking self-attention mechanism, the number of chunks in the text modality needs to be consistent with that in the speech modality. Therefore, a smaller Chunk Size is selected. Assume that the chunk size of the speech modality is W , the length of the current batch after zero padding is , the number of blocks is X , , the longest feature sequence length in the text modality is ,like The remainder of is 0, then the length of the text in the current batch is , otherwise the alignment length of the current batch is .

[0073] like Figure 2 As shown, in one embodiment, the text MCC-Transformer block is composed of 2 stacked MCC-Transformer blocks, with a total of 6 self-attention layers.

[0074] In some embodiments, the speech-text cross-MCC-Transformer block is composed of multiple MCC-Transformer blocks stacked together, and each MCC-Transformer block is composed of a multi-head self-attention C-MHSA module based on a uniform blocking mechanism, a multi-head self-attention SC-MHSA module based on a shift blocking mechanism, and a multi-head self-attention SSC-MHSA module based on a sequential sampling blocking mechanism.

[0075] The structures of the C-MHSA module, SC-MHSA module, and SSC-MHSA module are the same as those used in the encoder and will not be repeated here.

[0076] Specifically, if Figure 2 As shown, in one embodiment, the speech-text cross-MCC-Transformer block is composed of 2 stacked MCC-Transformer blocks, with a total of 6 self-attention layers.

[0077] Since the decoding end needs to input both acoustic signals and supervised text data during the training phase, and the length of the acoustic sequence signal is usually longer than that of the text sequence signal, in order to ensure the effectiveness of the cross-modal calculation of the self-attention mechanism, the text modality and the acoustic modality have the same number of blocks, and the block size of the text modality should not be too large, while too small blocks will not limit the context. Therefore, in the embodiment of the present application, on the premise of ensuring the block size as much as possible, the end of the text sequence is padded by zero padding to ensure uniform block size.

[0078] The speech recognition method provided by the present invention introduces a speech MCC-Transformer block formed by stacking multiple MCC-Transformer blocks in an encoder, each MCC-Transformer block is sequentially cascaded by a multi-head self-attention C-MHSA module based on a uniform blocking mechanism, a multi-head self-attention SC-MHSA module based on a shift blocking mechanism, and a multi-head self-attention SSC-MHSA module based on a sequential sampling blocking mechanism. Not only is the complexity of the encoder linear, but the complexity of the decoder is also linear, that is, the overall complexity is linear, so the overall training and prediction are more efficient, the graphics card memory occupancy is lower, long sequence model training and reasoning can be better supported, and the accuracy of speech recognition is improved.

[0079] In some embodiments, the encoder of the MCC-Transformer model further includes a front-end processing network, wherein the front-end processing network includes an input layer (not shown in the figure), a padding layer, a convolutional layer, a linear mapping layer, and a dropout layer;

[0080] The input layer is used to obtain the feature sequence;

[0081] The padding layer is used to fill zeros in the feature sequence;

[0082] The convolution layer is used to transform the dimension of the feature sequence;

[0083] The linear mapping layer is used to remap the feature sequence;

[0084] The dropout layer is used to set the outputs of some neurons to zero.

[0085] Specifically, if Figure 3 As shown in the figure, the principle of padding the feature sequence with zeros is: let chunksize be W , the longest feature sequence length in the current batch is L , then to ensure that the new sequence length is divisible by chunksize, if L / W The remainder of is 0, then the alignment length of the current batch is L; otherwise the alignment length of the current batch is .

[0086] The convolution layer is usually composed of a convolutional network with a convolution kernel of 3×3 and a stride of 2. After reducing the dimension of the feature sequence, the length of the feature sequence is 1 / 4 of the original length.

[0087] The linear mapping layer remaps the feature sequence after dimensionality reduction, mainly expanding its dimension.

[0088] During the training process, the dropout layer randomly sets the outputs of some neurons to zero. For example, the random dropout ratio is 0.1, that is, the outputs of neurons accounting for 0.1 are set to zero.

[0089] The speech recognition method provided by the present invention introduces a speech MCC-Transformer block formed by stacking multiple MCC-Transformer blocks in an encoder, each MCC-Transformer block is sequentially cascaded by a multi-head self-attention C-MHSA module based on a uniform blocking mechanism, a multi-head self-attention SC-MHSA module based on a shift blocking mechanism, and a multi-head self-attention SSC-MHSA module based on a sequential sampling blocking mechanism, thereby improving the accuracy of speech recognition.

[0090] The following is an explanation of the training process of the MCC-Transformer model:

[0091] First, divide the dataset into training set, validation set and test set.

[0092] Before model training, the annotated corpus needs to be further divided into training set, validation set and test set. Taking the example dataset AISHELL-1 as an example, the training set is about 150 hours, the validation set is about 10 hours, and the test set is about 5 hours.

[0093] In some embodiments, the training data may also be enhanced.

[0094] In order to make the trained model more robust, the embodiment of the present application uses variable rate enhancement and spectral enhancement (SpecAug) methods to enhance the training set data.

[0095] The training data is enhanced by variable rate, and three data are generated from the original 1.0 times data, whose rates are 0.9, 1.0, and 1.1 times, respectively. That is, during the training process, the specific rate of a certain speech in the current batch is determined to be 0.9, 1.0, or 1.1 with the same probability. The specific generation method can be a pre-generation method or an online conversion method.

[0096] Perform spectral enhancement on the training data. For example, the specific parameters of the SpecAug algorithm are selected as the frequency domain masking parameters.F =10, time domain mask parameter T =50.

[0097] Then, a vocabulary is generated based on the annotated data.

[0098] The annotated text of all the speech in the training set is segmented into "characters", and different "characters" are added to the vocabulary in sequence.

[0099] Added "unk", "blank", and "sos / eos" to the vocabulary.

[0100] Then, the original feature sequence is extracted from the training data. For example, an 80-dimensional FBank acoustic feature is used, including framing, windowing, Fourier transform, cepstral coefficient calculation, etc., with a frame width of 25 milliseconds and a frame shift of 10 milliseconds.

[0101] Then, the annotation of the input text sequence is calculated, the CTC loss is calculated, the loss score of the CTC part is calculated, and it is back-propagated together with the attention decoder loss to iteratively update the model parameters. The CTC decoder is a regularization for the self-attention decoder and can provide streaming output results in streaming scenarios.

[0102] Calculate the loss of the CTC decoder and the loss of the self-attention decoder Afterwards, the overall loss is calculated based on this Among them, the loss of the self-attention decoder is the cross entropy loss, and the output of the encoder is used as the key to the self-attention calculation ( K ) and databases ( V ), the output of the text MCC-Transformer block is taken as Q , are jointly input into the speech-text cross-MCC-Transformer block, the result is output in a self-encoding manner, and the overall loss is calculated after conversion by the Softmax layer , the overall loss The calculation formula is as follows:

[0103]

[0104] in, is a hyperparameter. For example, it can be set to 0.3 during the training phase and 0.5 during the prediction phase.

[0105] Finally, the original speech feature sequence and text sequence are input into the MCC-Transformer model for training, error backpropagation is performed based on the loss on the training set, the model parameters are updated, and the loss on the validation set is used to determine when to stop training the model.

[0106] Usually, if the loss function starts to show a monotonically increasing trend or a long-term oscillation trend on the validation set, you can consider stopping the training. In this example, the training round is stopped after 180 rounds, and a detection point model is saved at the end of each round of training.

[0107] After the model training is completed, the model performance is verified based on the validation set and compared with the performance of the existing model. The verification results on AISHELL-1 are shown in Table 1.

[0108] Table 1 Performance comparison of different models on AISHELL-1

[0109]

[0110] In Table 1, Transformer refers to a speech recognition model based on a conventional Transformer, C-Transformer is a speech recognition model based on a uniform blocking mechanism Transformer, U2-Transformer is a speech recognition model based on a time-constrained mechanism Transformer, SC-Transformer is a speech recognition model based on a shift blocking mechanism Transformer, SSC-Transformer is a speech recognition model based on a sequential sampling blocking mechanism Transformer, MCC-Transformer is a speech recognition model provided by this application, and the experimental results are experimental results using 2 GTX 3090s. It can be seen from the above experimental results that MCC-Transformer is significantly superior to the C-Transformer model that only uses a uniform blocking mechanism. MCC-Transformer is significantly superior to the two advanced Transformer models based on the blocking mechanism, SC-Transformer and SSC-Transformer, and can surpass the conventional Transformer model and U2-Transformer in performance. In addition, in terms of training time and real-time rate, MCC-Transformer maintains the advantages of the model based on the blocking mechanism.

[0111] The speech recognition device provided by the present invention is described below. The speech recognition device described below and the speech recognition method described above can be referred to each other.

[0112] Figure 4 is a structural diagram of a speech recognition device based on a multi-configuration block context converter model provided by the present invention, such as Figure 4 As shown, the speech recognition device provided by the present invention includes:

[0113] The acquisition module 401 is used to acquire a feature sequence of the speech to be recognized;

[0114] The recognition module 402 is used to input the feature sequence into a trained speech recognition model, perform speech recognition on the speech to be recognized using the speech recognition model, and obtain a recognition result output by the speech recognition model;

[0115] The speech recognition model is a multi-configuration block context transformer MCC-Transformer model. The MCC-Transformer model is constructed by introducing a speech MCC-Transformer block in the encoder on the basis of the transformer model. The speech MCC-Transformer block is stacked by multiple MCC-Transformer blocks, and each MCC-Transformer block is cascaded in sequence by a multi-head self-attention C-MHSA module based on a uniform blocking mechanism, a multi-head self-attention SC-MHSA module based on a shift blocking mechanism, and a multi-head self-attention SSC-MHSA module based on a sequential sampling blocking mechanism.

[0116] Specifically, the above-mentioned speech recognition device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned speech recognition method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0117] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the speech recognition method, which includes:

[0118] Obtaining a feature sequence of speech to be recognized;

[0119] Inputting the feature sequence into a trained speech recognition model, performing speech recognition on the speech to be recognized using the speech recognition model, and obtaining a recognition result output by the speech recognition model;

[0120] The speech recognition model is a multi-configuration block context transformer MCC-Transformer model. The MCC-Transformer model is constructed by introducing a speech MCC-Transformer block in the encoder on the basis of the transformer model. The speech MCC-Transformer block is stacked by multiple MCC-Transformer blocks, and each MCC-Transformer block is cascaded in sequence by a multi-head self-attention C-MHSA module based on a uniform blocking mechanism, a multi-head self-attention SC-MHSA module based on a shift blocking mechanism, and a multi-head self-attention SSC-MHSA module based on a sequential sampling blocking mechanism.

[0121] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0122] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can perform the speech recognition method provided by the above methods, the method comprising:

[0123] Obtaining a feature sequence of speech to be recognized;

[0124] Inputting the feature sequence into a trained speech recognition model, performing speech recognition on the speech to be recognized using the speech recognition model, and obtaining a recognition result output by the speech recognition model;

[0125] The speech recognition model is a multi-configuration block context transformer MCC-Transformer model. The MCC-Transformer model is constructed by introducing a speech MCC-Transformer block in the encoder on the basis of the transformer model. The speech MCC-Transformer block is stacked by multiple MCC-Transformer blocks, and each MCC-Transformer block is cascaded in sequence by a multi-head self-attention C-MHSA module based on a uniform blocking mechanism, a multi-head self-attention SC-MHSA module based on a shift blocking mechanism, and a multi-head self-attention SSC-MHSA module based on a sequential sampling blocking mechanism.

[0126] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the speech recognition method provided by the above methods, the method comprising:

[0127] Obtaining a feature sequence of speech to be recognized;

[0128] Inputting the feature sequence into a trained speech recognition model, performing speech recognition on the speech to be recognized using the speech recognition model, and obtaining a recognition result output by the speech recognition model;

[0129] The speech recognition model is a multi-configuration block context transformer MCC-Transformer model. The MCC-Transformer model is constructed by introducing a speech MCC-Transformer block in the encoder on the basis of the transformer model. The speech MCC-Transformer block is stacked by multiple MCC-Transformer blocks, and each MCC-Transformer block is cascaded in sequence by a multi-head self-attention C-MHSA module based on a uniform blocking mechanism, a multi-head self-attention SC-MHSA module based on a shift blocking mechanism, and a multi-head self-attention SSC-MHSA module based on a sequential sampling blocking mechanism.

[0130] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0131] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0132] "Determine B based on A" in the embodiments of the present application means that the factor A should be considered when determining B. It is not limited to "B can be determined based on A alone", but should also include: "Determine B based on A and C", "Determine B based on A, C and E", "Determine C based on A, and further determine B based on C", etc. In addition, it can also include taking A as a condition for determining B, for example, "When A meets the first condition, use the first method to determine B"; for another example, "When A meets the second condition, determine B", etc.; for another example, "When A meets the third condition, determine B based on the first parameter", etc. Of course, it can also be a condition that takes A as a factor for determining B, for example, "When A meets the first condition, use the first method to determine C, and further determine B based on C", etc.

[0133] In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech recognition method based on a multi-configuration block context transformer model, characterized in that: include: Obtaining a feature sequence of speech to be recognized; Inputting the feature sequence into a trained speech recognition model, performing speech recognition on the speech to be recognized using the speech recognition model, and obtaining a recognition result output by the speech recognition model; The speech recognition model is a multi-configuration block context transformer MCC-Transformer model. The MCC-Transformer model is constructed by introducing a speech MCC-Transformer block in the encoder on the basis of the transformer model. The speech MCC-Transformer block is stacked by multiple MCC-Transformer blocks, and each MCC-Transformer block is cascaded in sequence by a multi-head self-attention C-MHSA module based on a uniform blocking mechanism, a multi-head self-attention SC-MHSA module based on a shift blocking mechanism, and a multi-head self-attention SSC-MHSA module based on a sequential sampling blocking mechanism.

2. The speech recognition method based on the multi-configuration block context transformer model according to claim 1, characterized in that: The C-MHSA module includes a C-MHSA layer, and the C-MHSA layer is used to perform attention calculation within the C-MHSA module using uniform partitioning; The SC-MHSA module includes a SC-MHSA layer, and the SC-MHSA layer is used to perform attention calculation in the SC-MHSA module divided by shift blocks; The SSC-MHSA module includes an SSC-MHSA layer, which is used to perform attention calculation within the SSC-MHSA module using sequential sampling repartitioning.

3. The speech recognition method based on the multi-configuration block context transformer model according to claim 1, characterized in that: The decoder of the MCC-Transformer model includes a CTC decoder and a self-attention decoder; The self-attention decoder includes an input layer, an embedding layer, a text MCC-Transformer block, and a speech-text cross MCC-Transformer block; The input layer is used to obtain the previous output result of the decoder; The Embedding layer is used to contextualize the previous output result of the decoder; The text MCC-Transformer block is used to perform a mask operation on the output information of the Embedding layer; The speech-text cross MCC-Transformer block is used to calculate a feature matrix based on the output information of the encoder and the output information of the text MCC-Transformer block.

4. The speech recognition method based on the multi-configuration block context transformer model according to claim 3, characterized in that: The text MCC-Transformer block is composed of multiple MCC-Transformer blocks stacked together, and each MCC-Transformer block is cascaded in sequence by a multi-head self-attention C-MHSA module based on a uniform blocking mechanism, a multi-head self-attention SC-MHSA module based on a shift blocking mechanism, and a multi-head self-attention SSC-MHSA module based on a sequential sampling blocking mechanism.

5. The speech recognition method based on the multi-configuration block context transformer model according to claim 3, characterized in that: The speech-text cross-MCC-Transformer block is composed of multiple stacked MCC-Transformer blocks, and each MCC-Transformer block is cascaded in sequence by a multi-head self-attention C-MHSA module based on a uniform blocking mechanism, a multi-head self-attention SC-MHSA module based on a shift blocking mechanism, and a multi-head self-attention SSC-MHSA module based on a sequential sampling blocking mechanism.

6. The speech recognition method based on the multi-configuration block context transformer model according to any one of claims 1 to 5, characterized in that: The encoder of the MCC-Transformer model also includes an input layer, a padding layer, a convolutional layer, a linear mapping layer, and a dropout layer; The input layer is used to obtain the feature sequence; The padding layer is used to fill zeros in the feature sequence; The convolution layer is used to transform the dimension of the feature sequence; The linear mapping layer is used to remap the feature sequence; The dropout layer is used to set the outputs of some neurons to zero.

7. A speech recognition device based on a multi-configuration block context transformer model, characterized in that: include: An acquisition module, used to acquire a feature sequence of the speech to be recognized; A recognition module, used to input the feature sequence into a trained speech recognition model, perform speech recognition on the speech to be recognized using the speech recognition model, and obtain a recognition result output by the speech recognition model; The speech recognition model is a multi-configuration block context transformer MCC-Transformer model. The MCC-Transformer model is constructed by introducing a speech MCC-Transformer block in the encoder on the basis of the transformer model. The speech MCC-Transformer block is stacked by multiple MCC-Transformer blocks, and each MCC-Transformer block is cascaded in sequence by a multi-head self-attention C-MHSA module based on a uniform blocking mechanism, a multi-head self-attention SC-MHSA module based on a shift blocking mechanism, and a multi-head self-attention SSC-MHSA module based on a sequential sampling blocking mechanism.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the speech recognition method based on the multi-configuration block context transformer model is implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech recognition method based on the multi-configuration block context transformer model as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the speech recognition method based on the multi-configuration block context transformer model as described in any one of claims 1 to 6 is implemented.

Images