Speech recognition method based on multi-scale residual convolution and Longform
By combining multi-scale residual convolutional neural network and Longformer model, the problem of insufficient long-term dependence modeling of traditional speech recognition methods in long-sequence speech signal processing is solved, and the accuracy and computing efficiency of speech recognition are improved.
Patent Information
- Application Number
- CN202411612729.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-13
- Publication Date
- 2025-02-28
AI Technical Summary
Traditional speech recognition methods have problems such as insufficient modeling, disappearance or explosion of gradients, and low computational efficiency when processing long-sequence speech signals, resulting in a decrease in accuracy during complex background noise and long speech sequence recognition.
The speech recognition method based on multi-scale residual convolution and Longformer is adopted to extract multi-level features through multi-scale residual convolution neural network, and the Longformer model is used to capture long-term dependence context information.
It effectively solves the problem of insufficient long-term dependence modeling in traditional methods in long-sequence speech signal processing, improves the accuracy of speech recognition, significantly reduces computing overhead, and improves real-time processing capabilities.
Smart Images
Figure CN119541460A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of speech recognition, and in particular to a speech recognition method based on multi-scale residual convolution and Longformer. Background Art
[0002] With the development of artificial intelligence and deep learning technologies, speech recognition technology has been widely used in various application scenarios, such as smart assistants, smart homes, autonomous driving, and meeting records. The core of speech recognition technology is to convert speech signals into text. It not only needs to accurately recognize the speech content, but also needs to be robust in complex background noise, different speaker characteristics, and different contexts. However, traditional speech recognition methods have certain limitations in recognizing complex scenes and long speech sequences. In particular, when there is background noise and long-term dependency problems in the speech signal, the accuracy rate tends to drop significantly.
[0003] Traditional speech recognition models are usually based on a combination of Hidden Markov Model (HMM) and Gaussian Mixture Model (GMM), or combined with Deep Neural Network (DNN) for acoustic modeling. However, these methods have certain shortcomings in modeling long time series and capturing global context information. In recent years, Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) have been widely used in the field of speech recognition. Among them, CNN is good at extracting local features, while RNN is suitable for processing sequence data. However, these methods have the following problems when processing long speech signals:
[0004] Although RNN models (such as LSTM and GRU) can capture long-term dependency information, they are prone to gradient vanishing or gradient exploding problems when inputting long sequences, resulting in poor modeling results. Traditional convolutional neural networks rely on fixed-size convolution kernels for feature extraction, which makes it difficult to simultaneously capture feature information at different scales, resulting in poor results when processing signals containing multiple speech features. When faced with long-sequence speech signals, RNN models have low computational efficiency and are difficult to meet real-time processing requirements. Summary of the invention
[0005] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a speech recognition method based on multi-scale residual convolution and Longformer.
[0006] In order to achieve the above object, the main technical solutions adopted by the present invention include:
[0007] The embodiment of the present invention provides a speech recognition method based on multi-scale residual convolution and Longformer, comprising:
[0008] S1, preprocessing the pre-acquired speech signal to obtain a corresponding acoustic signal;
[0009] S2, inputting the acoustic signal into a trained speech recognition model to obtain a speech text corresponding to the speech signal;
[0010] The trained speech recognition model is obtained by training a pre-created speech recognition model using a training data set;
[0011] The training data set includes: a plurality of first acoustic signals and pre-annotated speech texts corresponding to each first acoustic signal; the plurality of first acoustic signals are obtained by pre-processing the plurality of speech signals obtained in advance for training;
[0012] The speech recognition model is constructed by a multi-scale residual convolutional neural network and a longformer model.
[0013] Preferably, before S1, it also includes:
[0014] S0. Use the training data set to train the pre-created speech recognition model to obtain a trained speech recognition model.
[0015] Preferably,
[0016] The acoustic signal is an 80-dimensional Fbank feature signal.
[0017] Preferably, it is characterized in that
[0018] The multi-scale residual convolutional neural network includes, in sequence: a first convolutional layer, a first branch structure, a first residual layer, a second convolutional layer, a second branch structure layer, a second residual layer, and a third convolutional layer;
[0019] Wherein, the first convolution layer is used to extract corresponding basic features from the acoustic signal input into the multi-scale residual convolutional neural network;
[0020] The first branch structure layer is used to use a plurality of pre-set convolution kernels to perform convolution processing on the basic features respectively, and concatenate the convolution processing results obtained by all the convolution kernels in the first branch structure layer to obtain a first comprehensive feature;
[0021] A first residual layer is used to perform residual processing on the first comprehensive feature and the basic feature to obtain a first residual result;
[0022] A second convolutional layer, used to perform convolution processing on the first residual result to obtain an intermediate feature;
[0023] The second branch structure layer is used to use a plurality of pre-set convolution kernels to perform convolution processing on the intermediate respectively, and splice the convolution processing results obtained by all the convolution kernels in the second branch structure layer to obtain a second comprehensive feature;
[0024] A second residual layer is used to perform residual processing on the second comprehensive feature and the intermediate feature to obtain a second residual result;
[0025] The third convolutional layer is used to perform convolution processing on the second residual result to obtain output features of the multi-scale residual convolutional neural network.
[0026] Preferably, the first branch structure layer includes: a 1×1 convolution kernel, a 3×3 convolution kernel, a 3×3 expansion convolution kernel, and a 5×5 expansion convolution kernel;
[0027] The second branch structure layer includes: a 1×1 convolution kernel, a 3×3 convolution kernel, a 3×3 expansion convolution kernel, and a 5×5 expansion convolution kernel.
[0028] Preferably, the longformer model is used to process the output features of the multi-scale residual convolutional neural network to obtain corresponding speech text.
[0029] Preferably, the longformer model comprises: N longformer encoders and M longformer decoders;
[0030] Among them, N longformer encoders are used to extract the global context information of the output features of the multi-scale residual convolutional neural network;
[0031] The M longformer decoders are used to generate corresponding speech text according to decoding the global context information.
[0032] Preferably, the S0 specifically includes:
[0033] The speech recognition model is trained using each first acoustic signal in the training data set and the corresponding pre-annotated speech text until a preset regression loss function corresponding to the speech recognition model converges to obtain a trained speech recognition model.
[0034] Preferably,
[0035] The AdamW optimization algorithm is used in the process of training the speech recognition model to minimize the preset regression loss function and update all weight parameters in the speech recognition model.
[0036] Preferably,
[0037] Among them, N is equal to 9; M is equal to 6.
[0038] The beneficial effects of the present invention are:
[0039] The present invention discloses a speech recognition method based on multi-scale residual convolution and Longformer, which combines a multi-scale residual convolution neural network and a Longformer model. The multi-scale residual convolution network can extract multi-level features of speech signals through convolution kernels of different scales, and can capture short-term local features and long-term global features in speech. The Longformer model can capture long-term dependent contextual information by using its local window attention mechanism and global attention mechanism. The combination of the two effectively solves the problem of insufficient long-term dependency modeling in traditional methods when processing long-sequence speech signals, thereby improving the accuracy of speech recognition.
[0040] The speech recognition model used in the speech recognition method based on multi-scale residual convolution and Longformer of the present invention introduces the Longformer model to replace the traditional recurrent neural network (RNN) or standard Transformer model. Longformer adopts a local window attention mechanism, which is more efficient in processing long sequence speech signals, can significantly reduce the computational overhead of the model while ensuring the recognition accuracy, and improves the real-time processing capability of the model.
[0041] The speech recognition model used in the speech recognition method based on multi-scale residual convolution and Longformer of the present invention combines the multi-scale residual convolution neural network with the Longformer, making full use of the advantages of both, so that the speech recognition model can extract fine local acoustic features and capture contextual information in long time series. This complementarity greatly improves the overall performance of the speech recognition system. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 A flow chart of a speech recognition method based on multi-scale residual convolution and Longformer in the present invention;
[0043] Figure 2 A schematic diagram of the structure of a multi-scale residual convolutional neural network in a speech recognition method based on multi-scale residual convolution and Longformer in the present invention;
[0044] Figure 3 This is a flow chart of a speech recognition method based on multi-scale residual convolution and Longformer in Embodiment 2 of the present invention. DETAILED DESCRIPTION
[0045] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.
[0046] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.
[0047] Embodiment 1
[0048] See also Figure 1 This embodiment provides a speech recognition method based on multi-scale residual convolution and Longformer, including:
[0049] S1, preprocessing the pre-acquired speech signal to obtain a corresponding acoustic signal;
[0050] The acoustic signal is an 80-dimensional Fbank feature signal.
[0051] S2, inputting the acoustic signal into a trained speech recognition model to obtain a speech text corresponding to the speech signal;
[0052] The trained speech recognition model is obtained by training a pre-created speech recognition model using a training data set;
[0053] The training data set includes: a plurality of first acoustic signals and pre-annotated speech texts corresponding to each first acoustic signal; the plurality of first acoustic signals are obtained by pre-processing the plurality of speech signals obtained in advance for training;
[0054] The speech recognition model is constructed by a multi-scale residual convolutional neural network and a longformer model.
[0055] In the practical application of this embodiment, before S1, the following steps are also included:
[0056] S0. Use the training data set to train the pre-created speech recognition model to obtain a trained speech recognition model.
[0057] For details, see Figure 2 , the multi-scale residual convolutional neural network includes in sequence: a first convolutional layer, a first branch structure, a first residual layer, a second convolutional layer, a second branch structure layer, a second residual layer, and a third convolutional layer;
[0058] Wherein, the first convolution layer is used to extract corresponding basic features from the acoustic signal input into the multi-scale residual convolutional neural network;
[0059] The first branch structure layer is used to use a plurality of pre-set convolution kernels to perform convolution processing on the basic features respectively, and concatenate the convolution processing results obtained by all the convolution kernels in the first branch structure layer to obtain a first comprehensive feature;
[0060] The first branch structure layer includes: a 1×1 convolution kernel, a 3×3 convolution kernel, a 3×3 expansion convolution kernel, and a 5×5 expansion convolution kernel;
[0061] In this embodiment, the first branch structure layer uses a variety of convolution kernels of different sizes, each of which has a different size and can capture acoustic features in different time ranges. Small-sized convolution kernels can capture fine-grained, short-term change features; large-sized convolution kernels can capture long-term global features. This multi-scale design makes up for the limitations of single-scale convolution kernels in feature extraction, so that the network has multi-level and multi-angle feature capture capabilities. Among them, the 3×3 dilated convolution kernel increases the receptive field and is used to capture features in a medium time range. The 5×5 dilated convolution kernel further expands the receptive field and is used to extract long-time scale features and capture the overall speech pattern. The combination of convolution kernels of different sizes can extract features at different time scales. The 1×1 convolution kernel is good at capturing fine-grained local features, and the 3×3 and 5×5 dilated convolution kernels can capture longer-term dependencies, giving the model multi-scale analysis capabilities. By splicing the convolution results of different convolution kernels, a comprehensive feature vector is formed, which retains more levels and angles of information and improves the expressive power of features.
[0062] The introduction of multi-scale convolution kernels enables the network to extract features of speech signals at different time scales, including short-term fine-grained features (such as instantaneous spectrum changes) and long-term overall features (such as phoneme changes). This enables the model to have a more comprehensive understanding of the temporal structure and spectral characteristics of speech signals, thereby improving the accuracy of speech recognition.
[0063] A first residual layer is used to perform residual processing on the first comprehensive feature and the basic feature to obtain a first residual result;
[0064] The first residual layer adds the output features of the first branch structure layer to the basic features of the first convolutional layer to obtain a first residual result.
[0065] The residual connection uses a jump structure to allow the gradient to propagate more smoothly from the deep network to the shallow network, effectively alleviating the gradient vanishing problem in the deep convolutional network. After the basic features are added to the comprehensive features, the basic feature information is retained, and at the same time, the higher-level multi-scale features are combined to improve the model's ability to represent input data. The residual structure makes the network training more stable and can achieve better convergence in fewer training iterations.
[0066] A second convolutional layer, used to perform convolution processing on the first residual result to obtain an intermediate feature;
[0067] The second convolutional layer performs convolution processing on the output of the first residual layer to further extract intermediate features. This layer is equivalent to deep refinement based on the first layer features. Through further convolution operations, higher-level abstract features such as phoneme features and pitch changes can be extracted to enhance the model's ability to parse complex speech signals. The extraction of intermediate features enables the network to capture more subtle feature changes, which helps the multi-scale convolution processing of the subsequent branch structure layer.
[0068] The second branch structure layer is used to use a plurality of pre-set convolution kernels to perform convolution processing on the intermediate respectively, and splice the convolution processing results obtained by all the convolution kernels in the second branch structure layer to obtain a second comprehensive feature;
[0069] The second branch structure layer includes: a 1×1 convolution kernel, a 3×3 convolution kernel, a 3×3 expansion convolution kernel, and a 5×5 expansion convolution kernel.
[0070] The second branch structure layer is similar to the first branch structure layer. The second branch structure layer performs multi-scale convolution processing at a deeper level, making feature extraction more detailed and comprehensive. The outputs of different convolution kernels are concatenated to form the second comprehensive feature, which contains more time scale information and improves the modeling ability of complex speech structures.
[0071] A second residual layer is used to perform residual processing on the second comprehensive feature and the intermediate feature to obtain a second residual result;
[0072] The second residual layer adds the output features of the second branch structure layer to the intermediate features of the second convolutional layer to obtain the second residual result. The residual connection enables the network to better retain the intermediate feature information, improve the generalization performance of the model, and adapt to different types of speech signal input. Through the residual connection, the second comprehensive feature is combined with the intermediate feature, retaining the multi-level information extracted previously and enhancing the richness of the feature representation.
[0073] The third convolutional layer is used to perform convolution processing on the second residual result to obtain output features of the multi-scale residual convolutional neural network.
[0074] In this embodiment, the third convolutional layer performs convolution processing on the second residual result to obtain the final output features of the multi-scale residual convolutional neural network. Through the last layer of convolution, the network further extracts high-level abstract features, providing richer and more representative input features for the subsequent Longformer model. The final output features contain a complete information chain from basic features to deep features, providing a reliable data foundation for long sequence modeling.
[0075] Specifically, the multi-scale residual convolutional neural network in this embodiment realizes comprehensive feature extraction from low-level to high-level and from short-time to long-time through the combination of multi-scale convolution kernels and residual connections. The multi-scale convolution kernel covers different time scales, making the feature extraction richer. The residual structure effectively alleviates the gradient vanishing problem and improves the training efficiency and stability of the model. The branch structure and residual connection reduce redundant calculations and improve the computational efficiency of the network. Through deep-level, multi-scale feature extraction and fusion, high-quality input features are provided for the subsequent Longformer model, which improves the performance and applicability of the overall speech recognition method.
[0076] In this embodiment, the longformer model is used to process the output features of the multi-scale residual convolutional neural network to obtain corresponding speech text.
[0077] The longformer model includes: N longformer encoders and M longformer decoders;
[0078] Among them, N longformer encoders are used to extract the global context information of the output features of the multi-scale residual convolutional neural network;
[0079] The M longformer decoders are used to generate corresponding speech text according to decoding the global context information.
[0080] Among them, N is equal to 9; M is equal to 6.
[0081] In this embodiment, the Longformer encoder layer number N = 9, the decoder layer number M = 6, this configuration makes a trade-off in balancing model performance and computational complexity. The encoder layer number N = 9 ensures that the model has sufficient depth to extract global context information. The decoder layer number M = 6 reduces the computational overhead and improves the reasoning efficiency of the model while maintaining the decoding performance. This design achieves a good balance between feature extraction depth and text decoding efficiency, is suitable for processing long sequence speech signals, and can run efficiently when computing resources are limited. The configuration of 9 encoder layers and 6 decoder layers can fully extract the global features of speech and generate accurate speech text, improving the overall performance of the speech recognition system.
[0082] In the speech recognition model, the two hyperparameters of the Longformer model, N (number of encoders) and M (number of decoders), are crucial to the performance and computational efficiency of the model. In order to find the best configuration, a configuration of 9 encoder layers and 6 decoder layers was selected. This configuration strikes a balance between performance and computational efficiency, can fully extract the features of the multi-scale residual convolutional network, and use the Longformer model to capture global context information, thereby improving the accuracy of speech recognition. N = 9, a sufficient number of encoder layers, can capture the global context features of speech signals. M = 6, with a moderate increase in the number of decoder layers, can provide more sophisticated speech-to-text decoding capabilities, suitable for complex scenarios and long speech inputs.
[0083] The S0 specifically includes:
[0084] The speech recognition model is trained using each first acoustic signal in the training data set and the corresponding pre-annotated speech text until a preset regression loss function corresponding to the speech recognition model converges to obtain a trained speech recognition model.
[0085] The AdamW optimization algorithm is used in the process of training the speech recognition model to minimize the preset regression loss function and update all weight parameters in the speech recognition model.
[0086] In this embodiment, the training data set includes a large amount of first acoustic signals (i.e., speech audio data) and their corresponding pre-annotated speech texts (i.e., manually annotated speech transcription texts). These data are used for supervised training, that is, the model uses known input-output pairs for learning during the training process. The acoustic signal and the corresponding text are used as input-output pairs and input into the speech recognition model for training. The model repeatedly adjusts its internal parameters so that it can predict the correct speech text from the input acoustic signal.
[0087] The loss function is used to measure the error between the model's predicted output (the recognized text sequence) and the actual annotated text sequence. In this embodiment, CTC (Connectionist Temporal Classification) loss or cross entropy loss is used as the regression loss function. The model is trained through multiple iterations. When the value of the loss function no longer decreases significantly, it is called loss function convergence. At this point, the parameters of the model have reached a better state, the training process ends, and the final trained speech recognition model is obtained.
[0088] Embodiment 2
[0089] See also Figure 3 This embodiment provides a speech recognition method based on multi-scale residual convolution and Longformer, including:
[0090] A1. Use the training data set to train the pre-created speech recognition model to obtain a trained speech recognition model.
[0091] The A1 specifically includes: using each first acoustic signal in the training data set and the corresponding pre-annotated speech text to train the speech recognition model until a preset regression loss function corresponding to the speech recognition model converges to obtain a trained speech recognition model. In the process of training the speech recognition model, the AdamW optimization algorithm is used to minimize the preset regression loss function and update all weight parameters in the speech recognition model.
[0092] A2. Preprocessing the pre-acquired speech signal to obtain a corresponding acoustic signal;
[0093] The acoustic signal is an 80-dimensional Fbank feature signal.
[0094] A3, inputting the acoustic signal into a trained speech recognition model to obtain a speech text corresponding to the speech signal;
[0095] The training data set includes: a plurality of first acoustic signals and pre-annotated speech texts corresponding to each first acoustic signal; the plurality of first acoustic signals are obtained by pre-processing the plurality of speech signals obtained in advance for training;
[0096] The speech recognition model is constructed by a multi-scale residual convolutional neural network and a longformer model.
[0097] See also Figure 2The multi-scale residual convolutional neural network includes: a first convolutional layer, a first branch structure, a first residual layer, a second convolutional layer, a second branch structure layer, a second residual layer, and a third convolutional layer in sequence; wherein the first convolutional layer is used to extract the corresponding basic features of the acoustic signal input into the multi-scale residual convolutional neural network; the first branch structure layer is used to use a plurality of pre-set convolutional kernels to perform convolution processing on the basic features respectively, and splice the convolution processing results obtained by all the convolution kernels in the first branch structure layer to obtain the first comprehensive feature; the first branch structure layer includes: a 1×1 convolution kernel, a 3×3 convolution kernel, a kernel, 3×3 dilated convolution kernel, 5×5 dilated convolution kernel; the first residual layer is used to perform residual processing on the first comprehensive feature and the basic feature to obtain a first residual result; the second convolution layer is used to perform convolution processing on the first residual result to obtain an intermediate feature; the second branch structure layer is used to use a plurality of pre-set convolution kernels to perform convolution processing on the intermediate respectively, and splice the convolution processing results obtained by all convolution kernels in the second branch structure layer to obtain a second comprehensive feature; the second branch structure layer includes: 1×1 convolution kernel, 3×3 convolution kernel, 3×3 dilated convolution kernel, 5×5 dilated convolution kernel. The second residual layer is used to perform residual processing on the second comprehensive feature and the intermediate feature to obtain a second residual result; the third convolution layer is used to perform convolution processing on the second residual result to obtain the output feature of the multi-scale residual convolution neural network.
[0098] In the second embodiment, the first convolution layer uses a one-dimensional convolution (1D Convolution) and performs convolution along the time frame (time series) direction. The first convolution layer is used to map the acoustic signal into a 128-dimensional basic feature, which can increase the expressive power of the feature and help the subsequent network extract more useful acoustic information.
[0099] The 1×1 convolution kernel in the first branch structure layer processes the basic features to obtain the first convolution result F1, where F1=W 1×1 *X0; X0 is the basic feature; W 1×1 is the weight matrix of the 1×1 convolution kernel; * is the convolution operation.
[0100] The 3×3 convolution kernel in the first branch structure layer processes the basic features to obtain the first convolution result F2, where F2=W 3×3 *X0;W 3×3 is the weight matrix of the 3×3 convolution kernel;
[0101] The 3×3 dilated convolution kernel in the first branch structure layer processes the basic features to obtain the first convolution result F3, where F3=W 3×3 dil *X0;W 3×3dil is the weight matrix of the 3×3 dilated convolution kernel;
[0102] The 1×1 convolution kernel in the first branch structure layer processes the basic features to obtain the first convolution result F4, where F4=W 5×5 dil *X0;W 5×5 dil is the weight matrix of the 5×5 dilated convolution kernel;
[0103] The first comprehensive feature obtained at the first branch structure layer is F concat ;
[0104] F concat =Concat(F1, F2, F3, F4); Concat() is a concatenation operation. In this embodiment, all features extracted by different convolution kernels are retained through the concatenation operation, so that the comprehensive features contain multi-scale and multi-angle information. The dimension of the comprehensive features after concatenation increases, so that the model can process more feature information in subsequent layers, and improve the modeling ability of complex speech signals.
[0105] In this embodiment, an improved residual updating method is proposed for the first residual layer, specifically, formula (1) is used to perform residual operation;
[0106] The formula (1) is:
[0107] R = a·F concat +(1-a)·X0;
[0108] Among them, R is the feature matrix output by the first residual layer after residual processing, that is, the first residual result;
[0109] a is a dynamic weight parameter with a value range of [0, 1], which is used to control the weight ratio of comprehensive features to input features and is dynamically calculated using formula (2);
[0110] The formula (2) is:
[0111]
[0112] σ( ) is the sigmoid activation function;
[0113] F concat [j] represents the activation value of the j-th dimension feature in the 128-dimensional basic features;
[0114] The dynamic weight parameter a is automatically adjusted according to the importance of the feature, giving different weights to the comprehensive features and basic features, making the model more flexible when fusing features. By dynamically adjusting a, the model can adapt to different input feature distributions and maintain high recognition accuracy when processing different speech signals. The residual connection structure can help the gradient pass from the deep network to the shallow network, avoiding the common gradient vanishing phenomenon in the deep convolutional network and improving the training efficiency and effect. Formula (2) enables formula (1) to automatically adjust the feature fusion ratio according to the importance of the feature, thereby improving the robustness and adaptability of the model.
[0115] The longformer model is used to process the output features of the multi-scale residual convolutional neural network to obtain the corresponding speech text. The longformer model includes: N longformer encoders and M longformer decoders; wherein the N longformer encoders are used to extract the global context information of the output features of the multi-scale residual convolutional neural network; and the M longformer decoders are used to generate the corresponding speech text according to decoding the global context information. Wherein, N is equal to 9; M is equal to 6.
[0116] In the second embodiment, during the training of the speech recognition model, the learning rate is dynamically adjusted according to formula (3);
[0117] The formula (3) is:
[0118]
[0119] γ t is the learning rate of the tth iteration;
[0120] γ0 is the preset initial learning rate;
[0121] is the gradient norm of the tth iteration, which represents the gradient size of the regression loss function L to the speech recognition model parameters.
[0122] is the gradient norm of the t-1th iteration.
[0123] In the second embodiment, the learning rate can be adjusted dynamically according to the gradient change through formula (3), so that the convergence speed of the model can be improved. When the gradient changes greatly, the learning rate increases to help the model jump out of the local optimal solution; when the gradient changes slightly, the learning rate decreases to prevent oscillation and overfitting. By automatically adjusting the learning rate according to the gradient change, the model can converge more smoothly during the training process, reducing the problem of training instability caused by improper learning rate setting.
[0124] The solution in Example 2 combines a multi-scale residual convolutional neural network and a Longformer model. Through carefully designed convolutional layers, residual structures, dynamic feature fusion mechanisms, and adaptive optimization algorithms, an efficient, robust, and adaptable speech recognition system is constructed. In Example 2, convolution kernels of various sizes (1×1, 3×3, 3×3 expansion, 5×5 expansion) are used, and parallel convolution processing is performed in a branch structure. By performing multi-scale convolution operations on the input features, acoustic features at different time scales are captured, and the fusion of fine-grained and long-term dependent features is achieved. After multi-scale feature extraction, the solution combines the outputs of different convolution kernels into a feature matrix through splicing operations, thereby improving the richness and diversity of feature expression.
[0125] Convolution kernels of different sizes can capture feature information in different time ranges in speech signals. 1×1 convolution kernels are used for information fusion and dimensionality reduction, 3×3 convolution kernels are used to capture local detail features, and dilated convolution kernels are used to extract long-term dependency features. By concatenating the outputs of different convolution kernels, the solution retains all feature information, allowing the network to process richer inputs in subsequent layers, improving the accuracy and generalization ability of the speech recognition model.
[0126] In addition, the residual structure and dynamic weight adjustment mechanism are introduced in the second embodiment, and the AdamW optimization algorithm is adopted to effectively alleviate the gradient vanishing problem of the deep network and improve the training efficiency and stability of the model. The dynamic learning rate adjustment strategy enables the model to adaptively change the learning rate, avoid falling into the local optimum, and help the model converge quickly. The residual structure can retain the information of shallow features in the deep network through jump connections, so that the gradient can be smoothly propagated, thereby avoiding the gradient disappearance and improving the training efficiency of the model.
[0127] AdamW introduces a weight decay mechanism to avoid overfitting problems caused by excessive weights, and combines momentum and adaptive learning rate adjustment to improve the robustness of training. Adjusting the learning rate according to gradient changes can speed up convergence, reduce the risk of oscillation and overfitting, and make the model more stable during training.
[0128] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0129] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0130] In the present invention, unless otherwise clearly specified and limited, when a first feature is “on” or “below” a second feature, it may be that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. Moreover, when a first feature is “above”, “above” or “above” a second feature, it may be that the first feature is directly above or obliquely above the second feature, or it may simply mean that the first feature is higher in level than the second feature. When a first feature is “below”, “below” or “below” a second feature, it may be that the first feature is directly below or obliquely below the second feature, or it may simply mean that the first feature is lower in level than the second feature.
[0131] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.
[0132] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may alter, modify, replace and modify the above embodiments within the scope of the present invention.
Claims
1. A speech recognition method based on multi-scale residual convolution and Longformer, characterized in that: include: S1, preprocessing the pre-acquired speech signal to obtain a corresponding acoustic signal; S2, inputting the acoustic signal into a trained speech recognition model to obtain a speech text corresponding to the speech signal; The trained speech recognition model is obtained by training a pre-created speech recognition model using a training data set; The training data set includes: a plurality of first acoustic signals and pre-annotated speech texts corresponding to each first acoustic signal; the plurality of first acoustic signals are obtained by pre-processing the plurality of speech signals obtained in advance for training; The speech recognition model is constructed by a multi-scale residual convolutional neural network and a longformer model.
2. The speech recognition method based on multi-scale residual convolution and Longformer according to claim 1, characterized in that: Prior to S1, it also included: S0. Use the training data set to train the pre-created speech recognition model to obtain a trained speech recognition model.
3. The speech recognition method based on multi-scale residual convolution and Longformer according to claim 2, characterized in that: in, The acoustic signal is an 80-dimensional Fbank feature signal.
4. The speech recognition method based on multi-scale residual convolutional neural network and Longformer according to claim 2, characterized in that: The multi-scale residual convolutional neural network includes, in sequence: a first convolutional layer, a first branch structure, a first residual layer, a second convolutional layer, a second branch structure layer, a second residual layer, and a third convolutional layer; Wherein, the first convolution layer is used to extract corresponding basic features from the acoustic signal input into the multi-scale residual convolutional neural network; The first branch structure layer is used to use a plurality of pre-set convolution kernels to perform convolution processing on the basic features respectively, and concatenate the convolution processing results obtained by all the convolution kernels in the first branch structure layer to obtain a first comprehensive feature; A first residual layer is used to perform residual processing on the first comprehensive feature and the basic feature to obtain a first residual result; A second convolutional layer, used to perform convolution processing on the first residual result to obtain an intermediate feature; The second branch structure layer is used to use a plurality of pre-set convolution kernels to perform convolution processing on the intermediate respectively, and splice the convolution processing results obtained by all the convolution kernels in the second branch structure layer to obtain a second comprehensive feature; A second residual layer is used to perform residual processing on the second comprehensive feature and the intermediate feature to obtain a second residual result; The third convolution layer is used to perform convolution processing on the second residual result to obtain output features of the multi-scale residual convolutional neural network.
5. The speech recognition method based on multi-scale residual convolution and Longformer according to claim 4, characterized in that: The first branch structure layer includes: a 1×1 convolution kernel, a 3×3 convolution kernel, a 3×3 expansion convolution kernel, and a 5×5 expansion convolution kernel; The second branch structure layer includes: a 1×1 convolution kernel, a 3×3 convolution kernel, a 3×3 expansion convolution kernel, and a 5×5 expansion convolution kernel.
6. The speech recognition method based on multi-scale residual convolution and Longformer according to claim 5, characterized in that: The longformer model is used to process the output features of the multi-scale residual convolutional neural network to obtain corresponding speech text.
7. The speech recognition method based on multi-scale residual convolution and Longformer according to claim 6, characterized in that: The longformer model includes: N longformer encoders and M longformer decoders; Among them, N longformer encoders are used to extract the global context information of the output features of the multi-scale residual convolutional neural network; The M longformer decoders are used to generate corresponding speech text according to decoding the global context information.
8. The speech recognition method based on multi-scale residual convolution and Longformer according to claim 7, characterized in that: The S0 specifically includes: The speech recognition model is trained using each first acoustic signal in the training data set and the corresponding pre-annotated speech text until a preset regression loss function corresponding to the speech recognition model converges to obtain a trained speech recognition model.
9. The speech recognition method based on multi-scale residual convolution and Longformer according to claim 8, characterized in that: The AdamW optimization algorithm is used in the process of training the speech recognition model to minimize the preset regression loss function and update all weight parameters in the speech recognition model.
10. The speech recognition method based on multi-scale residual convolution and Longformer according to claim 8, characterized in that: in, N equals 9; M equals 6.