Speech recognition system of convolutional enhanced self-attention model based on improved scoring matrix

Abstract

The invention discloses a speech recognition system based on a convolutional enhanced self-attention model of an improved scoring matrix. In the invention, a new relation calculation module (RCM) is introduced, so that the attention recovery decoding method is optimized. The module aims to mitigate error accumulation in an attention recovery decoding method, thereby reducing the influence of CTC decoding errors on subsequent attention recovery. In addition, the system also introduces a jump fusion module (SFM) to integrate shallow and deep features. The limitation that a Conformer encoder lacks inter-layer skipping connection is solved, and the capability of effectively capturing and utilizing contextual information of the model is enhanced. Comparison experiments show that a graph-based method improves the recognition rate, especially on a relatively small Uygur language data set. Compared with a baseline model, the method has the advantages that the character error rates are respectively reduced by 0.03%, 0.35% and 0.44%. On a GeneralSpeech16.1 Uygur language data set, the character error rate of the method is 4.19%, and is reduced by 1.56% compared with that of a baseline model.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech recognition, and in particular to a speech recognition system based on a convolution-enhanced self-attention model with an improved scoring matrix. Background Art

[0002] Speech recognition technology is the technology that converts speech signals into a format that computers can recognize. It can be used in a variety of applications, including voice assistants, voice search, speech-to-text annotation, and translation. Traditional speech recognition methods rely on acoustic features and language models to match speech signals with predefined patterns. However, they struggle with complex speech signals and unstable voice changes, and require a large number of artificial features, limiting their generalization capabilities across different speech signals. Subsequently, hidden Markov models model the speech signal and combine it with pronunciation dictionaries and language models for recognition.

[0003] The Conformer-Transformer model is a commonly used architecture for speech recognition. During training, a joint loss consisting of CTC and attention is adopted. In addition, the inference process utilizes Attention Rescoring (AR).

[0004] But its Conformer encoder is more complex and only focuses on deep features. During the decoding process of the AR decoder, many incorrectly decoded results are input into the attention model. During the autoregressive process, the attention model accumulates errors, leading to the error accumulation problem. Summary of the invention

[0005] The object of the present invention is to provide a speech recognition system based on a convolution-enhanced self-attention model with an improved scoring matrix in order to solve the above-mentioned problems.

[0006] The technical solution adopted by the present invention is as follows: a speech recognition system based on a convolution-enhanced self-attention model with an improved scoring matrix, the system comprising an encoder and a decoder;

[0007] The encoder is internally configured with: a Conformer encoder, a jump fusion module, and an internal loss module;

[0008] The decoder is internally provided with: a CTC decoder and a relationship calculation module;

[0009] The Conformer encoder: as a basic encoder, extracts speech features and captures long-term dependencies in speech signals;

[0010] The jump fusion module fuses the outputs of Conformer encoders from different levels, combines shallow and deep features, and enhances the model's ability to capture contextual information.

[0011] The internal loss module calculates the difference between the downsampled data and the original data and incorporates it into the model loss for optimization, thereby reducing information loss during the downsampling process and speeding up training;

[0012] The CTC decoder decodes the features output by the encoder to generate a rough text sequence;

[0013] The relationship calculation module scores the CTC decoding results and optimizes the scoring matrix using differential technology and regularization methods to reduce error accumulation in the attention re-scoring process and improve decoding performance.

[0014] In a preferred embodiment, the jump fusion module fuses shallow features and deep features by using an inter-layer jump connection method, fusing the outputs of Conformers from layers 5, 8, and 11. This is because the Conformer layers at shallow depths contain more noise, which in turn will have an adverse effect on the recognition results.

[0015] In a preferred embodiment, the jump connection in the Conformer encoder model exists in the Conformer layer, and there is no jump connection between the Conformer layers of the Conformer encoder. The shallow layer contains more detailed information, while the deep layer contains more semantic information.

[0016] In a preferred embodiment, the internal loss module performs downsampling using a convolution operation with a step size of 2 and performs upsampling using bilinear interpolation. The specific operation involves replacing the original pooling downsampling operation with a convolution operation. This is because convolution is a parameterized operation that allows subsequent optimization, while pooling is a parameterless operation that cannot be further optimized. The internal loss module uses bilinear interpolation for upsampling to minimize the number of parameters.

[0017] In a preferred embodiment, during use of the internal loss module, the output OL2 from the second layer of the Conformer undergoes downsampling by a convolution operation with a stride of 2;

[0018] It then undergoes an upsampling operation using bilinear interpolation, resulting in an output OL2'. The difference between OL2 and OL2', called the inner loss, is calculated and added to the final loss for joint optimization. As training progresses, the convolution operation with a stride of 2 learns the best downsampling result that minimizes information loss. This approach to inner loss reduces the input length, speeds up training, and minimizes information loss.

[0019] In a preferred embodiment, the internal loss module chooses to use the output from the second layer of Conformer as the starting layer for calculating the internal loss because the earlier coding layer may contain more noise. Calculating the internal loss too early may lead to unstable training. The InLoss of this article does not require a decoding process. Calculate the MSE (mean square error) loss, such as formula (1), formula (2):

[0020]

[0021] InLoss=L(Upsample2(Conv2(OL2)),OL2) (2);

[0022] Conv2 represents a 1D convolution with a stride of 2. Upsample2 represents an upsampling with a stride of 2. Finally, InLoss is added to the final loss for unified optimization.

[0023] In a preferred embodiment, the relationship calculation module first subtracts the numbers on the diagonal from the original score matrix to obtain a difference matrix. This step weakens the association between each position and itself, because the diagonal numbers usually represent the strongest association with each position. While considering global information, the system can focus on information from other positions through the difference matrix. Next, the difference matrix is ​​regularized to ensure that the score of each position falls within a specific range, and finally an attention score is generated. After regularization, the attention score obtained can better balance the degree of association between each position and other positions. This enhanced method improves the accuracy and reliability of RCM when calculating attention.

[0024] In a preferred embodiment, the regularization process of the relationship calculation module is called regularization control. The calculation formula is formula (3), in which, when the value of S is small, the highest point of the function is low, resulting in a low score along the diagonal of the score matrix. On the other hand, when the value of R is small, the distribution range of the score becomes wider, indicating that the inhibitory effect on the score matrix is ​​small. D represents the difference matrix.

[0025] Score=S(sigmod(R*D)*(1-sigmod(R*D))) (3).

[0026] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0027] 1. In the present invention, the SFM module fuses shallow and deep features, so that the model can capture the information in the speech signal more comprehensively, thereby improving the recognition accuracy. The RCM module optimizes the scoring matrix, reduces the error accumulation in the attention re-scoring process, and makes the decoding result more accurate. InLoss module: By calculating the difference between the downsampled data and the original data, and incorporating it into the model loss for optimization, it reduces the information loss in the downsampling process, thereby speeding up the training speed. InLoss module: effectively reduces the number of model parameters, reduces the complexity of the model, and facilitates model deployment and application. Each module has independent functions and is easy to expand and optimize. For example, more effective attention fusion methods can be explored, or more efficient downsampling methods can be studied.

[0028] 2. In the present invention, the system has good scalability and can be extended to speech recognition of other languages ​​by adjusting model parameters or adding language models. The system can be applied to various speech recognition scenarios, including voice assistants, voice search, voice input methods, voice translation, etc., to provide users with a more convenient voice interaction experience.

[0029] 3. In the present invention, the attention recovery decoding method is optimized by introducing a new relational computation module (RCM). This module is designed to alleviate the error accumulation in the attention recovery decoding method, thereby reducing the impact of CTC decoding errors on subsequent attention recovery. In addition, the system also introduces a skip fusion module (SFM) to integrate shallow and deep features. This solves the limitation of the lack of inter-layer skip connections in the Conformer encoder and enhances the model's ability to effectively capture and utilize contextual information. Comparative experiments show that the graph-based method improves the recognition rate, especially on relatively small Uyghur datasets. On the Aishell1, Primewords, and ST datasets, the method reduces the character error rate by 0.03%, 0.35%, and 0.44%, respectively, compared with the baseline model. On the GeneralSpeech16.1 Uyghur dataset, the word error rate of this method is 4.19%, which is 1.56% lower than the baseline model. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is the overall working flow chart of the present invention;

[0031] Figure 2 The figure is a calculation process diagram of the score matrix in the present invention; wherein Sc represents the original score matrix, and Rc represents the new score matrix. The values ​​on the diagonal of the new score matrix are the hyperparameters S;

[0032] Figure 3The RC function diagram of the present invention. When the input is 0, the output is the hyperparameter S of the system, which represents the maximum value of the score matrix. S controls the amplitude of the autocorrelation, while R controls the aperture size of the function, thereby controlling the correlation between the values ​​on the diagonal matrix and other positions;

[0033] Figure 4 Verification loss diagram in the present invention;

[0034] Figure 5 It is an error analysis diagram in the present invention;

[0035] Figure 6 It is the experimental parameter setting diagram of the present invention;

[0036] Figure 7 This is a comparison experiment diagram between the present invention and the SOTA model. Prime refers to the Primewords dataset;

[0037] Figure 8 This is the ablation study diagram in the present invention. ECMISM-RCM represents the experimental results after removing RCM from ECMISM. The others follow the same logic. Time refers to the training time. Prime refers to the PrimeWords dataset;

[0038] Fig. 9 It is the influence diagram of S and R parameters on RCM in the present invention;

[0039] Fig.10 It is an experimental data diagram of the RCM module in the present invention, where AV refers to the average number of models used in decoding;

[0040] Fig.11 This is a diagram for verifying the effectiveness of SFM in the present invention;

[0041] Fig.12 This is a diagram for confirming the effectiveness of InLoss in the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] Reference Figure 1-12 ,

[0044] Example:

[0045] This system uses a conformer as an encoder, and based on it, a skip fusion module (SFM) is designed to fuse shallow and deep features. In addition, in order to reduce information loss during downsampling and speed up training, this system introduces the InLoss module. In the decoder part, this system proposes a new relational computation module (RCM) to solve the problem of error accumulation in the attention recovery decoding method. The overall workflow of the model is as follows Figure 1 shown.

[0046] Figure 1 In the figure, DC2 refers to the downsampling convolution with a sampling rate of 2, and UP2 refers to the up-pooling with a sampling rate of 2. OL0 represents the output of layer 0, and the same applies to other layers. EC refers to the output of the shared encoder. During training, the output of the shared encoder is jointly decoded with the CTC and REM decoders. The dotted line in the figure represents the inference process. During inference, the CTC decoding result is input and fed to the RCM for re-decoding. When the InLoss module is not used, the output shape of each encoder layer remains consistent. Specifically, the size of OL3-11 is the same as that of OL0-2.

[0047] The jump fusion module includes:

[0048] The jump connection in the Conformer model mainly exists in the Conformer layer, and there is no jump connection between the Conformer layers. The shallow layer contains more detailed information, while the deep layer contains more semantic information. Therefore, this system proposes an inter-layer jump connection method to fuse shallow and deep features. This method can speed up the convergence of the model and achieve good recognition performance. This system adopts an additive average fusion method. Figure 1 In Figure 2, we can observe that the system fuses the outputs of Conformers from layers 5, 8, and 11. This is because Conformer layers at shallower depths contain more noise, which in turn adversely affects the recognition results.

[0049] The internal loss module includes:

[0050] There are many ways to optimize the Conformer model. The most direct way is to downsample the input length. Reducing the input sequence length can effectively reduce the training time. However, if you only rely on downsampling to reduce the length of the speech input, it will inevitably lead to information loss, and this loss is irreversible.

[0051] Therefore, our system uses convolution operations with a stride of 2 to perform downsampling, and bilinear interpolation to perform upsampling. The specific operation involves replacing the original pooling downsampling operation with a convolution operation. This is because convolution is a parameterized operation that allows subsequent optimization, while pooling is a parameterless operation that cannot be further optimized. Our method uses bilinear interpolation for upsampling to minimize the number of parameters.

[0052] In a particular process, the output OL2 from the second layer of the Conformer undergoes downsampling with a convolution operation with stride 2. Subsequently, it undergoes an upsampling operation using bilinear interpolation, resulting in output OL2'. The difference between OL2 and OL2', called the inner loss, is calculated and added to the final loss for joint optimization. As training progresses, the convolution operation with stride 2 learns the best downsampling result, thereby minimizing information loss. This approach to inner loss reduces the input length, speeds up training, and minimizes information loss.

[0053] The choice of using the output from the second layer of the Conformer as the starting layer for calculating the internal loss is due to the possibility that earlier coding layers may contain more noise. Calculating the internal loss too early may lead to training instability. The InLoss of this system does not require a decoding process. This system calculates the MSE (mean square error) loss, as shown in formula (1), formula (2):

[0054]

[0055] InLoss=L(Upsample2(Conv2(OL2)),OL2) (2)

[0056] Conv2 represents a 1D convolution with a stride of 2. Upsample2 represents an upsampling with a stride of 2. Finally, InLoss is added to the final loss for unified optimization.

[0057] The relational calculation module includes:

[0058] In the Wenet toolkit, an AR decoding method is adopted, in which the decoding results of CTC are re-scored by the attention model to achieve improved decoding results, because the decoding results of CTC beam search involve selecting the top n possible results as the input attention model. When the input contains a large number of erroneous characters, the decoding results of the attention model are unfavorable. Therefore, RCM is introduced in this system. RCM mainly improves the calculation method of the scoring matrix, such as Figure 2As shown. The algorithm first subtracts the numbers on the diagonal from the original score matrix to obtain a difference matrix. This step weakens the association of each position with itself, because the diagonal numbers usually represent the strongest association with each position. While considering global information, the system can pay attention to information from other positions through the difference matrix. Next, the difference matrix is ​​regularized to ensure that the score of each position falls within a specific range, and finally an attention score is generated. After regularization, the resulting attention score can better balance the degree of association between each position and other positions. This enhanced method improves the accuracy and reliability of RCM when calculating attention.

[0059] The regularization process is called regularization control (RC). Its function is as follows Figure 3 As shown, the calculation formula is expressed in Formula 3. In the formula, when the value of S is small, the highest point of the function is low, resulting in lower scores along the diagonal of the score matrix. On the other hand, when the value of R is small, the distribution range of the scores becomes wider, indicating less suppression of the score matrix. D represents the difference matrix.

[0060] Score=S(sigmod(R*D)*(1-sigmod(R*D))) (3)

[0061] 3 Results Analysis

[0062] In the experiment, this system used four datasets, three Chinese and one Uyghur dataset. The Chinese datasets used by this system are Aishell 1, Primewords Chinese Corpus Set 1 (Primewords) and FreeST Chinese Mandarin Corpus (ST). For the Uyghur dataset, this system uses Common Voice16.1. This system is called Ug in the experiment.

[0063] Aishell 1 contains about 178 hours of speech data. There are 120,000 samples for training, 14,000 for validation, and 7,000 for testing. Primewords is also a Chinese speech recognition dataset consisting of about 100 hours of audio data. However, unlike the Aishell 1 dataset, Primewords is not pre-partitioned into training, validation, and test sets. Based on the split ratio used in the Aishell 1 dataset, this system randomly splits the Primewords dataset into training, validation, and test sets in a ratio of 0.85:0.1:0.05. The total duration of the ST dataset is 109 hours and consists of 102,600 WAV files. Similarly, no split was performed, and this system divided the dataset in a ratio of 0.85:0.1:0.05. In the Ug dataset, 147 hours of validation data are available. The dataset already provides a test set and a validation set. This system merges all the remaining data into the training set.

[0064] On the Aishell, ST, and Ug datasets, the batch size is 36, while on the Primewords dataset, the batch size is 24. The hyperparameter settings for the three Chinese datasets remain consistent. However, due to language differences, the hyperparameter settings are slightly changed compared to the Uyghur dataset. Figure 6 The settings of other hyperparameters are listed. It must be emphasized that CTC_weight refers to the weight assigned to the CTC branch. During training, the weight of the CTC decoder is set to 0.3, while during inference, the CTC weight is adjusted to 0.5. These settings are consistent with the settings of the baseline system Wenet toolkit of this system.

[0065] The baseline system of this system is implemented using the Wenet toolkit. The standard CT model uses a downsampling rate of 4, called the CT_D4 model, while the CT model with a downsampling rate of 8 is called the CT_D8 model. In the experiment, this system uses CT_D4 and CT_D8 as the baseline system of this system, keeping the parameter settings consistent with Wenet. In this experiment, this system uses a server with 4 Tesla T4 GPUs, each with 16GB of memory capacity. The CPU used in the server is Intel Xeon Gold 5218R.

[0066] The evaluation indicators used in the experiment include character error rate (CER), word error rate (WER), and the training time of the model. In Chinese speech recognition, CER is easier to understand, while WER is affected by the segmentation system. Therefore, this system uses CER as the evaluation indicator for Chinese. However, for Uyghur, this system uses WER as the evaluation criterion. The smaller the CER and WER values, the better the performance. In the following table, CTC-Greedy, CTC-Beam, and Attention-rescoring represent the CER of the decoding strategies CTC Greedy Search, CTC Beam Search, and Attention-rescoring methods.

[0067] Comparative experiment

[0068] This system selected 9 commonly used speech recognition methods and conducted comparative experiments on 4 data sets. Among them, CT_D4 and CT_D8 are variants based on the Conformer_Transformer model, which are of great significance in speech recognition. The CT_D4 model uses a downsampling factor of 4 to reduce the input length to a quarter of the original size, while the downsampling rate of CT_D8 is 8. Speech Transformer and Transformer (Wenet) are both based on the Transformer model. Wenet's Transformer adopts a U-shaped architecture and combines Attention loss and CTC loss to improve speech recognition performance. BAT and RNNT are both based on the RNN-T model. RNNT combines acoustic information with historical output information, although the training cost is high, while BAT effectively reduces the training cost by restricting the search path. Efficient Conformer is an improved version of the Conformer model, which increases the training and inference speed and reduces the cost by adding downsampling in the middle layer of the Conformer. NST is a semi-supervised training method that includes training a series of models and using previously trained models to process unlabeled data to obtain corresponding labels for subsequent model training. In addition, these methods are compared with the large-scale Whisper model. Unfortunately, the Whisper version does not support Uyghur speech recognition.

[0069] Figure 7It is shown that compared with the baseline system CT_D4, the model of our system achieves 0.03%, 0.35%, 0.44% and 1.56% error rate reduction, respectively. The ECMISM model of our system significantly improves accuracy by reducing the CER of the Aishell1 dataset to 4.58%, approaching the state-of-the-art (SOTA) Conformer model (CER of 4.56%). The model of our system shows excellent performance on the Uyghur dataset, improving the performance of the baseline system by 1.56%. However, the improvement on the Aishell 1 dataset is relatively modest, which is attributed to the linguistic characteristics of Uyghur, which has a variety of word forms, resulting in increased errors during CTC decoding. The model of our system effectively mitigates this accumulated error. While CTC decoding performs well for Mandarin, the correction effect of our model is slightly less obvious, resulting in some improvement. Compared with the Transformer model, our system significantly enhances the encoding ability by using global and local information methods, reducing the error rate by 0.72%, 2.4%, 0.69% and 2.09% on the four datasets, respectively. Compared with the Efficient Conformer model, the method of this system corrects the accumulated errors in the CTC decoding process, thereby obtaining a superior decoding result, which significantly reduces the error rate by 0.97% on the Uyghur dataset. It is worth noting that whisper(base) performs the worst in these datasets, which is attributed to its multi-task nature. It performs well in processing multiple languages, but performs poorly in single-language tasks compared to single-task models.

[0070] 3.2 Ablation Experiment

[0071] from Figure 8 It can be clearly seen that after removing the RCM module, the error rate increased by 0.03 and 0.16, while removing the SFM module increased by 0.01 and 0.12, respectively. Removing both modules simultaneously resulted in an increase of 0.03 and 0.35. The results show that both the RCM module and the SFM module show significant improvements on the Prime dataset. On the Aishell 1 dataset, the SFM module has a relatively small effect. However, as Fig.11 As shown, the SFM module shows substantial advantages over the individual models. The SFM functions of the fusion model and this system have similarities in that both can integrate information across multiple layers. When the two modules are used together, better overall performance can be achieved. This is because SFM effectively fuses deep and shallow features, suppresses erroneous CTC decoding information after passing through the RCM module, and thus improves the decoding results. In addition, the ECMISM+InLoss model of this system reduces the training time by about 28%, while the accuracy is only reduced by 0.08% and 0.11%.

[0072] Module details experiment

[0073] In the experiments in this section, this system only shows the experimental results on the Aishell1 and Prime words datasets. The RCM model of this system introduces two parameters, S and R. This system conducts a series of extensive experiments to evaluate the impact of S and R parameters on the experimental results. The specific results are shown in Fig. 9 .

[0074] Fig. 9 The data in are the average of 80 model results. In the table, “None” indicates the case where the model does not converge. It can be observed that the best performance is achieved when S, R equals 1. Excessive increase or decrease in correlation will make the attention scores sparse, resulting in poor performance. In subsequent experiments, this system sets S and R to their optimal values, which are configured as 1.

[0075] Fig.10 The experimental data of the RCM module is shown. The value of CTC greedy search in the second row is 4.98%, which is slightly higher than the 4.94% reported in the Wenet paper. The Wenet code has been updated, resulting in a slight deviation in the experimental results. From the table, we can see that the RCM model of this system obtained a CER of 4.59% and 12.67% after averaging 80 models, exceeding Wenet's 4.61% and 12.90%. The reason for averaging 80 models is that after adding RCM, the later models in training become relatively stable, and the parameter differences between them are small. This can be seen in Figure 4 Without using the average model, i.e., when the average model is 1 in the first and fifth rows of the table, adding the RCM module of the present system improves the recognition accuracy by 0.26% and 0.54%.

[0076] Fig.11 The experimental data of the SFM module are shown. For comparison, the system Fig.10 Some data was extracted and placed in Fig.11 In. Compare Fig.10 and Fig.11, it can be observed that after adding the SFM module of this system, the best performance of the two datasets is 4.61 and 12.71 respectively. The performance of the Aishell1 dataset remains the same as the baseline, while the performance of the Primewords dataset is improved by 0.19. However, the experimental results of this system on individual models are significantly better than the baseline, with improvements of 0.39 and 0.88 respectively. This is because the fusion model also adopts the method of information fusion, which enhances the decoding performance. However, during the training process, the fusion model cannot be optimized. In contrast, the SFM of this system can be optimized during training. When used in combination with the RCM module, SFM can achieve superior results. This can also be seen in Figure 4 It can be observed in the loss shown, where after adding SFM, the convergence is faster and the loss is lower.

[0077] The simulation results verify the effectiveness of the model. Next, this system integrates the InLoss module into the model. Fig.12 The data in represent the experimental results on the Aishell 1 dataset. Fig.12 The data in show that the model using the InLoss method is significantly better than the CT_D8 model, especially when using beam search decoding. This also shows that the InLoss method has certain advantages in feature extraction. Compared with CT_D4, this model also significantly improves the training speed, saving about 28% of the training time.

[0078] Data Visualization:

[0079] In order to more easily prove the effectiveness of the model of this system, this system draws a verification loss graph on the Aishell1 dataset, such as Figure 4 As shown. The two black lines represent the baseline. The orange line in the figure represents the validation loss after incorporating the SFM module. It can be observed that the convergence speed of the model has been significantly improved, and the loss of the 75th batch is the lowest among all models. The blue line reflects the loss after integrating the RCM module, indicating that the loss is already lower than the baseline system after the 175th iteration, indicating the effectiveness of the RCM module. The ECMISM model of this system combines the advantages of the RCM and SFM modules, represented by the green line, showing the lowest loss. The red line represents the loss after incorporating the InLoss module, which appears to be the highest. This is because InLoss is added to the original loss. In fact, its recognition performance has only experienced a slight decline, 0.08% and 0.11% for the two datasets, respectively, while achieving a 28% increase in training speed.

[0080] In speech recognition, various errors can occur. Figure 5to analyze and gain insight into these errors. The data in the figure describes the statistics of various error types in the ST dataset. The test set of the ST dataset contains a total of 55,833 characters. In the figure, "S" represents substitution errors, which indicate recognition errors; "D" represents deletion errors, which indicate characters that are present in the label but missed by the model; and "I" represents insertion errors, which indicate characters that are recognized by the model but not originally present in the label. It can be observed that the overall error rate of the model of this system is significantly lower. Specifically, substitution and insertion errors show a significant reduction. However, there is no significant change in deletion errors. Deletion errors are usually caused during the encoding process of the encoder. The model of this system uses the same encoder as the baseline, resulting in limited improvements in this regard.

[0081] This application proposes an ECMISM speech recognition model based on the Conformer Transformer structure. In the encoder part, this system designs SFM to integrate shallow and deep features to obtain more information representation. In the decoder part, a new RCM algorithm is proposed to solve the error accumulation problem in attention recovery decoding, especially to reduce the impact of CTC decoding errors on subsequent attention recovery. Experimental results show that these two modules have a positive contribution to the speech recognition model. On the Aishell1, Prime and ST datasets, the CERs achieved by this system are 4.58%, 12.55% and 7.51%, respectively, which are lower than the baseline error rates of 4.61%, 12.90% and 7.95%. At the same time, this system achieves a WER of 4.19% on the General Voice 16.1 Uyghur dataset, which is lower than the baseline of 5.75%. In order to speed up the training speed, this system introduces the InLoss module. Integrating the InLoss module into the ECMISM model of this system can increase the training speed by 28%, and the error rates of the first two datasets are only increased by 0.08% and 0.11%, respectively. The InLoss module of this system is crucial to the lightweight design of the vehicle.

[0082] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0083] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech recognition system based on a convolution-enhanced self-attention model with an improved scoring matrix, characterized by: The system includes an encoder and a decoder; The encoder is internally configured with: a Conformer encoder, a jump fusion module, and an internal loss module; The decoder is internally provided with: a CTC decoder and a relationship calculation module; The Conformer encoder: as a basic encoder, extracts speech features and captures long-term dependencies in speech signals; The jump fusion module fuses the outputs of Conformer encoders from different levels, combines shallow and deep features, and enhances the model's ability to capture contextual information; The internal loss module calculates the difference between the downsampled data and the original data and incorporates it into the model loss for optimization, thereby reducing information loss during the downsampling process and speeding up training; The CTC decoder decodes the features output by the encoder to generate a rough text sequence; The relationship calculation module scores the CTC decoding results and optimizes the scoring matrix using differential technology and regularization methods to reduce error accumulation in the attention re-scoring process and improve decoding performance.

2. The speech recognition system of the convolution-enhanced self-attention model based on the improved scoring matrix as claimed in claim 1, characterized in that: The jump fusion module fuses shallow features and deep features through inter-layer jump connections, fusing the outputs of Conformers from the 5th, 8th and 11th layers.

3. The speech recognition system of the convolution-enhanced self-attention model based on the improved scoring matrix as claimed in claim 1, characterized in that: The skip connections in the Conformer encoder model exist within the Conformer layers, and there are no skip connections between the Conformer layers of the Conformer encoder; the shallow layers contain more detailed information, while the deep layers contain more semantic information.

4. The speech recognition system of the convolution-enhanced self-attention model based on the improved scoring matrix as claimed in claim 1, characterized in that: The internal loss module performs downsampling using a convolution operation with a stride of 2 and performs upsampling using bilinear interpolation; the specific operation involves replacing the original pooling downsampling operation with the convolution operation.

5. The speech recognition system of the convolution-enhanced self-attention model based on the improved scoring matrix as claimed in claim 1, characterized in that: During the use of the internal loss module, the output OL2 from the second layer of Conformer undergoes downsampling with a convolution operation with stride 2; It then undergoes an upsampling operation using bilinear interpolation to produce output OL2'; the difference between OL2 and OL2', called the internal loss, is calculated and added to the final loss for joint optimization; as training progresses, the convolution operation with a stride of 2 learns the optimal downsampling result.

6. The speech recognition system of the convolution-enhanced self-attention model based on the improved scoring matrix as claimed in claim 1, characterized in that: The internal loss module chooses to use the output from the second layer of Conformer as the starting layer for calculating the internal loss because the earlier encoding layers may contain more noise; calculating the internal loss too early may cause training instability; the InLoss in this paper does not require a decoding process; Calculate the MSE (mean square error) loss, as shown in formula (1), formula (2): InLoss=L(Upsample2(Conv2(OL2)),OL2) (2); Among them, Conv2 represents a 1D convolution with a stride of 2; Upsample2 represents an upsampling with a stride of 2; finally, InLoss is added to the final loss for unified optimization.

7. The speech recognition system of the convolution-enhanced self-attention model based on the improved scoring matrix as claimed in claim 1, characterized in that: The relationship calculation module first subtracts the numbers on the diagonal from the original score matrix to obtain a difference matrix; while considering global information, the system pays attention to information from other positions through the difference matrix; next, the difference matrix is ​​regularized to ensure that the score of each position falls within a specific range, and finally an attention score is generated; After regularization, the obtained attention scores better balance the degree of correlation between each position and other positions.

8. The speech recognition system of the convolution-enhanced self-attention model based on the improved scoring matrix as claimed in claim 1, characterized in that: The regularization process of the relationship calculation module is called regularization control; the calculation formula is formula (3), in which, when the value of S is small, the highest point of the function is low, resulting in a lower score along the diagonal of the score matrix; on the other hand, when the value of R is small, the distribution range of the score becomes wider, indicating that the suppression effect on the score matrix is ​​small; D represents the difference matrix; Score=S(sigmod(R*D)*(1-sigmod(R*D))) (3).

Images