Spoken English recognition method, system and device, medium and program product

Abstract

The invention discloses a spoken English recognition method, system and device, a medium and a program product, and relates to the technical field of speech recognition, and the method comprises the steps: extracting a time feature and a frequency feature from an audio feature; the dependency of the time dimension and the frequency dimension is calculated in parallel through dimension transformation on the time feature and the frequency feature, the time dependency feature and the frequency dependency feature are fused through a cross attention module, and a three-stage encoder is adopted to carry out progressive down-sampling processing; in the first stage, a grouping multi-head self-attention mechanism is adopted to carry out grouping calculation along the time dimension, and finally a speech recognition result is obtained through decoding, the time-frequency characteristics of speech features are concerned, and the model calculation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a method, system, device, medium and program product for English spoken recognition. Background Art

[0002] Speech recognition is a technology that converts speech signals into text or commands, also known as automatic speech recognition ( ). Speech recognition systems are able to convert a speaker's spoken speech into written text, enabling computers to understand and process speech input.

[0003] Before the rise of deep learning technology, speech recognition mainly used traditional signal processing and pattern matching technologies. Pattern matching technologies include dynamic time warping and hidden Markov models, which are used to model and match the characteristics of speech signals to recognize the speaker's voice.

[0004] However, traditional speech recognition methods have some significant disadvantages. First, such methods are usually based on limited vocabulary and hand-designed grammatical rules, which limit their ability to handle a wide vocabulary and freer language expressions. Second, such methods rely on hand-designed feature extraction methods, such as Mel-frequency cepstral coefficients and linear prediction coding, which require domain expertise and may not fully capture abstract features in speech signals. Finally, the models used by such methods, such as Gaussian mixture models and hidden Markov models, have certain limitations in capturing complex speech patterns and long-term dependencies.

[0005] Deep learning models can better handle complex speech patterns, larger data sets, and are more robust to environmental noise. However, although deep learning-based speech recognition methods have achieved remarkable results, there are still some shortcomings, such as the model mainly performs dependency calculations on the time dimension of audio data, and insufficient data volume will lead to poor actual recognition results of the model.

[0006] For example, a novel convolutional enhancement in speech recognition Architecture Since it can capture global and local features from audio signals simultaneously, Modules not only become It is a commonly used model for tasks and has also become a model for various end-to-end speech processing tasks. However, most of its model functions are allocated to time modeling, specifically the dependency calculation of the time dimension. Although speech is a time series signal, unlike text data, the characteristics of speech signals are in the time-frequency domain, so the frequency domain also plays another indispensable role.

[0007] In addition, the main driving force of deep neural networks is the availability of large amounts of training data, but this is not always feasible for speech recognition. This is because it is very expensive to collect large amounts of audio data with text labels, especially in oral exam scenarios, where most candidates' spoken pronunciation is not standard and uniform, and the audio contains a large portion of silence or intermittent segments, which greatly increases the cost of transcription. This type of low-quality audio data makes it difficult to train a high-accuracy speech recognition model, affecting recognition accuracy. Summary of the invention

[0008] In order to solve the above problems, the present invention proposes a method, system, device, medium and program product for English spoken recognition, which is designed (Convolutional Network Net Generation, next generation convolutional neural network) module stacking reinforcement The original downsampling module in the module focuses on the time-frequency characteristics of speech features. A time-frequency dependency calculation module is designed after the downsampling module to simultaneously calculate the dependencies of the time dimension and the frequency dimension and fuse the two features. At the same time, the progressive downsampling and grouped multi-head self-attention mechanism are integrated in the encoder to improve the calculation efficiency of the model.

[0009] In order to achieve the above object, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for spoken English recognition, comprising: Acquire speech signals and extract audio features, and extract time features and frequency features from the audio features; After the time feature is transformed in dimension, the time dependency is calculated, and the time dependency feature is obtained by residual connection with the transformed time feature. After the frequency feature is transformed in dimension, the frequency dependency is calculated, and the frequency dependency feature is obtained by residual connection with the transformed frequency feature. The time dependency feature and the frequency dependency feature are fused by cross attention mechanism and residual connection to obtain the fused feature. The fused features are progressively downsampled using a three-stage encoder. In the first stage, the fused features are grouped along the time dimension using a grouped multi-head self-attention mechanism, and attention is calculated separately for each group. The output features of the three-stage encoder are decoded to obtain the character probability distribution, and the speech recognition result is obtained based on it.

[0010] As an optional implementation, the process of extracting time features and frequency features from audio features includes: using stacked The modules constitute a downsampling module, and The channels of the module are increased successively to extract time features and frequency features; specifically: The audio features are processed by stride The convolutional layer performs downsampling in the time dimension and frequency dimension, and then inputs into the first In the module; First The output of the module is sequentially passed through the After the convolution layer and batch normalization operation, the input is In the module; second The output of the module is sequentially passed through the After the convolution layer and batch normalization operation, the input is In the module; third The output of the module is transformed in dimension through layer normalization and linear layer operations to extract time and frequency features.

[0011] As an optional implementation, the process of extracting time-dependent features and frequency-dependent features includes: reshaping the time features and frequency features respectively, and then using the time domain Block and frequency domain The blocks calculate the time domain dependency and frequency domain dependency respectively, and then unify them to the same dimension through the linear layer after residual connection, so as to obtain the time dependency feature and frequency dependency feature respectively.

[0012] As an optional implementation method, the feature fusion process includes: fusing the time-domain dependent features and the frequency-domain dependent features through a cross-attention mechanism, performing a residual connection with the input time features and frequency features after linear projection and dimensionality reshaping to obtain the fused features.

[0013] As an optional implementation, the grouped multi-head self-attention mechanism includes: using the grouped multi-head self-attention mechanism to group along the time dimension, the attention heads in each group share the same parameters, and after each group is linearly transformed, the obtained attention matrix is ​​sent to the attention heads in each group to calculate the attention, thereby reducing the attention complexity from Down to ,in, is the sequence length, is the feature dimension, is the number of groups, and the dimension of the attention matrix changes from × becomes × , , .

[0014] As an optional implementation, the English spoken language recognition method further includes: the speech recognition result is obtained by processing the trained speech recognition module, and the speech recognition module is trained using a pre-training-fine-tuning paradigm, wherein the parameters of the time-frequency dependency calculation process, the parameters in the first stage encoder and the parameters of the decoding process are fine-tuned, and uniformly adopted Activation function.

[0015] In a second aspect, the present invention provides an English spoken language recognition system, comprising: A feature extraction and downsampling module is configured to obtain a speech signal and extract audio features, and extract time features and frequency features from the audio features; The time-frequency dependency calculation module is configured to calculate the time dependency of the time feature after the dimension transformation, and perform residual connection with the time feature after the dimension transformation to obtain the time dependency feature, calculate the frequency dependency of the frequency feature after the dimension transformation, and perform residual connection with the frequency feature after the dimension transformation to obtain the frequency dependency feature, and fuse the time dependency feature and the frequency dependency feature by adopting the cross attention mechanism and residual connection to obtain the fused feature; The encoding module is configured to use a three-stage encoder to perform progressive downsampling on the fused features; in the first stage, the fused features are grouped along the time dimension using a grouped multi-head self-attention mechanism, and attention is calculated separately for each group; The decoding module is configured to decode the output features of the three-stage encoder to obtain character probability distribution, and thereby obtain speech recognition results.

[0016] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in the first aspect is performed.

[0017] In a fourth aspect, the present invention provides a computer-readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, the method described in the first aspect is performed.

[0018] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0019] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a spoken English recognition method that can be used in an examination environment, which has the following characteristics and advantages: (1) Innovative design Module stacking to strengthen The original downsampling module in the module is used to better utilize the time-frequency characteristics of the speech signal in the downsampling stage and extract richer high-dimensional features.

[0020] (2) An innovative time-frequency dependency calculation module is designed. The dependency of the time dimension and the frequency dimension is calculated in parallel by cleverly transforming the dimension. Then, the time-dependent features and the frequency-dependent features are fused through the cross-attention module. Then, a densely connected The Densely connected convolutional networks (Dense Convolutional Networks) module is used to solve the gradient vanishing problem and transfer more time-frequency domain information in the speech recognition model.

[0021] (3) It innovatively combines progressive downsampling and grouped multi-head self-attention mechanism. The encoder in the first stage adopts the grouped multi-head self-attention mechanism, which reduces the attention complexity from Down to ; The encoder in the later stage reduces the data length by half, thereby improving the encoder's computational efficiency and making it possible to deploy the speech recognition model on edge devices.

[0022] (4) Unify the activation functions of the convolution modules in the encoder and use them uniformly The activation function simplifies the speech recognition model without reducing the performance, making the deployment of the speech recognition model easier.

[0023] (5) The pre-training-fine-tuning paradigm was adopted to solve the problem of the high cost of obtaining internal datasets. When the internal dataset was severely insufficient, the speech recognition model was first trained using a public dataset with sufficient data volume, and then the internal dataset was used to fine-tune the parameters of the front part of the speech recognition model, which also achieved good recognition performance.

[0024] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0026] Figure 1 for The structure diagram of the module; Figure 2 for The structure diagram of the convolution module in the module; Figure 3The overall structure diagram of the speech recognition model provided in Example 1 of the present invention; Figure 4 The audio feature extraction flow chart provided in Example 1 of the present invention; Figure 5 A structural diagram of a downsampling module provided in Embodiment 1 of the present invention; Figure 6 A structural diagram of a time-frequency dependency calculation module provided in Example 1 of the present invention; Figure 7 A schematic diagram of progressive downsampling provided in Embodiment 1 of the present invention; Figure 8 A schematic diagram of grouped multi-head self-attention provided in Example 1 of the present invention; Fig. 9 This is a structural diagram of the convolution module in the speech recognition model provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0028] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0029] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "comprise" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0031] Example 1 At present, a novel convolution enhancement is proposed in the field of speech recognition Architecture Since it can capture global and local features from audio signals simultaneously, Modules not only become It is a commonly used model for tasks and has also become a model for various end-to-end speech processing tasks. The structure of the module is as follows Figure 1As shown, each The module contains two fully connected layers , in two In between, a multi-head self-attention module with four heads is used , followed by a convolutional module .

[0032] Its convolution module structure is as follows Figure 2 As shown, the layers are normalized, point-by-point convolutional layers, and Activation function to reduce the gradient vanishing problem. Then, input to The one-dimensional depth convolution layer of the activation function is then input into another point convolution layer. Finally, use The layer regularizes the network.

[0033] Specifically, for the input features , each The calculation process of the module is: (1); (2); (3); (4); in, represents a fully connected layer, whose output is multiplied by After input Add to get the intermediate features ; Represents a multi-head self-attention layer, whose output is the same as the input Add to get the intermediate features ; Represents a convolutional layer, whose output and input Add to get the intermediate features ;at last, After a fully connected layer Then multiply by Then input Add, and finally pass through a layer normalization layer Get the output .

[0034] Although the speech recognition method based on deep learning has achieved remarkable results, there are still some shortcomings, such as the model mainly performs dependency calculations on the time dimension of audio data, and insufficient data volume will lead to poor actual recognition effect of the model. Module, most of its model functions are assigned to time modeling, specifically the calculation of dependencies in the time dimension. Although speech is a time series signal, unlike text data, the characteristics of speech signals are in the time-frequency domain, so the frequency domain also plays another indispensable role.

[0035] Therefore, in order to pay more attention to the time-frequency characteristics of speech features and make the model more efficient and accurate, this embodiment proposes a new speech recognition model; first, the stacking of ConvNext (Convolutional Network NetGeneration, next generation convolutional neural network) modules is designed to strengthen Then, after the downsampling module, a time-frequency dependency calculation module is designed to simultaneously calculate the dependencies of the time dimension and the frequency dimension, and fuse the two features, and then input them into the densely connected In the Densely connected convolutional networks (Dense Convolutional Networks) module, the problem of gradient disappearance is solved, while more time-frequency domain information is transmitted in the network.

[0036] Among them, the above improvements to the downsampling module and the introduction of the time-frequency dependency calculation module inevitably increase the amount of calculation, which brings about the problem of model efficiency. In order to solve this problem, first of all, we improve the macro The overall architecture of the module introduces progressive downsampling In the module; at the same time, improve at the micro level The module's calculation method applies a grouped multi-head self-attention mechanism and unifies the selection of activation functions to further improve the model's calculation efficiency.

[0037] In addition, the main driving force behind powerful deep neural networks is the availability of large amounts of training data, which is not always feasible for speech recognition. This is because it is very expensive to collect large amounts of audio data with text labels, especially in oral exam scenarios, where most candidates’ spoken pronunciation is not standard and uniform, and the audio contains a large portion of silence or intermittent segments, which greatly increases the cost of transcription. This type of low-quality audio data makes it difficult to train a high-accuracy speech recognition model, which brings many inconveniences to further work after recognition.

[0038] In order to solve this problem, this embodiment adopts the pre-training-fine-tuning paradigm to train the speech recognition model, specifically: first, the speech recognition model is pre-trained on a public dataset to enable the speech recognition model to have a relatively good generalization ability, so that useful features can be extracted; then, some parameters of the speech recognition model are fine-tuned on a small number of internal examination room datasets, so that the speech recognition model can achieve a relatively good recognition accuracy rate for a small number of internal examination room datasets, thereby making up for the defect that it is not feasible to obtain a large number of transcribed internal examination room datasets.

[0039] like Figure 3 The figure shows the overall structure of the speech recognition model, whose input is audio features (Mel-frequency cepstral coefficients). First, the downsampling module is used to reduce the sequence length of the data, improve the computational efficiency of subsequent modules, and increase the model decoding speed; the time-frequency dependency calculation module is used to extract the time dependency and frequency dependency of the data in parallel, and fuse the two to obtain new features; the encoder is used to extract high-dimensional hidden features of the audio for the input of the decoder; the decoder recognizes the corresponding text sequence based on the high-dimensional hidden features output by the encoder.

[0040] The following is a detailed introduction to each module.

[0041] 1. Feature extraction module.

[0042] Mel cepstral coefficients ( ) is one of the commonly used feature extraction methods in speech signal processing and speech recognition. In this embodiment, MFCCs are used as input features of the speech recognition model.

[0043] Specifically, Figure 4 As shown: (1) A pre-emphasis filter is applied to the speech signal to amplify the high frequencies, thereby improving the signal-to-noise ratio of the speech signal and eliminating the effects of the vocal cords and lips during phonation, while avoiding numerical problems during the short-time Fourier transform operation.

[0044] (2) After pre-emphasis, the speech signal is divided into short-time frames by frame division and windowing. Because in most cases, the speech signal is non-stationary, it is meaningless to perform Fourier transform on the entire speech signal because the frequency contour of the speech signal will be lost over time. The speech signal is a short-time stationary signal, so it is necessary to perform Fourier transform on the short-time frame to obtain a good approximation of the signal frequency contour by connecting adjacent frames.

[0045] In this embodiment, the speech signal is framed as frame, The frame length of the speech signal is sampling points; at the same time, the frame shift is set to After the speech signal is divided into frames, each frame is multiplied by a window function to increase the continuity of the left and right ends of the frame, offset the assumption of the Fast Fourier Transform (FFT) that the data is infinite, and reduce spectral leakage.

[0046] The Hamming window has the form: (5); in, , is the window length, It is a parameter and can be set manually.

[0047] (3) Since it is usually difficult to see the characteristics of speech signals in the time domain, it is usually The energy distribution in the frequency domain is transformed to observe. Different energy distributions represent different speech characteristics. The short-time Fourier transform (FFT) is also called the short-time Fourier transform ( ). Then calculate the power spectrum, take the modulus square of the spectrum of the speech signal to get the spectral line energy P of the speech signal: (6); in, It is frame speech signal, N is the total number of frames.

[0048] (4) The power spectrum is passed through Filter Bank ( The filter bank is taken The triangular filter of the filter) is used to extract the frequency band and obtain The input features of dimension are finally obtained as , represents the number of samples in the training set ( ), Represents the time dimension, Represents the frequency dimension.

[0049] 2. Downsampling module .

[0050] Introduced in the downsampling module module to better extract time dimension features and frequency dimension features. ConvNext is a deep residual network ( )'s Convolutional Neural Networks (CNN) structure, and compared to, With a larger kernel size and depth convolution, it is more accurate and efficient. In this embodiment, the kernel size is set to 7.

[0051] like Figure 5 As shown in (a), the downsampling module It has a downsampling layer The stacking of modules is designed in three stages, and the channel size of each stage is gradually increased. For input frame rate Audio features , the processing process is: (7); (8); (9); (10); in, Representation layer normalization operation; Represents the operation of a linear layer; , , Represents the operations of the first convolutional layer, the second convolutional layer, and the third convolutional layer; Represents batch normalization operation; , , Indicates the first Module, second Module and third Operation of the module.

[0052] Specifically, for the input In the first stage (Equation (7)), we first pass the stride The convolutional layer Downsampling is performed in the time and frequency dimensions to reduce the computational complexity of subsequent steps, and then batch normalization is used Perform stable training and then input Output from the module ; The second stage (Equation (8)) and the third stage (Equation (9)) do not perform downsampling operations, and set the convolution layer and The stride is , because downsampling here would hurt the result, subsequent temporal downsampling is placed at the encoder In the example, the sampling rate is Finally, a linear layer is used Convert data dimensions to subsequent time-frequency dependency calculation modules The input dimension gets the output features . Assume that the input audio feature dimension is ,go through After the module, the output features The dimension becomes .

[0053] Through the above design, the downsampling module can simultaneously extract the time features and frequency features in the audio features, thereby giving full play to the time-frequency characteristics of the speech signal.

[0054] like Figure 5 As shown in (b), The module includes a depth convolution, two point convolutions, a normalization layer, and a activation layer and a scaling layer; where the deep convolution is two-dimensional, because the speech features have a time-frequency dimension, this embodiment introduces deep convolution to make the calculation more efficient; the role of the two point-by-point convolutions is similar to The feedforward submodule in the network ( ).

[0055] Specifically, the first Module as an example, for the input feature , the processing process is: (11); (12); (13); in, Represents the operation of a depth-wise separable convolutional layer; represents the first reshaping operation, which is used to transform the audio feature dimension from Reshape , represent , Represents the time dimension, represents the frequency dimension, represents the channel dimension; Represents a layer normalization operation. The output of the first reshape operation passes through a layer normalization layer to obtain the first output ; Represents the first point convolution operation, whose output channel is the input channel times; Represents the operation of the activation function. The output of the first point convolution operation is passed through The activation function gets the second output ; Represents the second point convolution operation, whose output channel is the input channel ; Represents the operation of the scaling layer; Represents the second reshape operation, which is used to change the data dimension from Reshape , and then with the input features Add to get the output .

[0056] 3. Time-frequency dependence calculation module .

[0057] Same as downsampling module Similarly, considering the time-frequency characteristics of the audio signal, the encoder should simultaneously calculate the time dependency and frequency dependency of the data. To this end, this embodiment innovatively The module designs a novel time-frequency dependent computation module , through the self-attention module in the Conformer block, the features containing time dimension dependencies can be calculated separately and frequency-dependent features ,like Figure 6 As shown, using the time domain Block and frequency domain Blocks are used to capture time domain dependency and frequency domain dependency respectively. Block and frequency domain The block structure uses Figure 1 The structure shown.

[0058] Specifically, given a containing both time features and frequency features , whose dimensions are , The calculation process of the module is as follows: (14); (15); (16); (17); in, Represents the time feature dimension reshaping operation; Represents the frequency feature dimension reshaping operation; Represents the operation of the time domain module; Represents the operation of the frequency domain module; Represents the operation of a linear projection layer; represents the cross-attention feature fusion operation; Represents the fusion feature dimension reshaping operation.

[0059] Input The dimensions are respectively and Reshaped into and , and then pass them through the time domain module and frequency domain modules The time-dependent and frequency-dependent features are calculated separately, and the input and output are residually connected. Finally, a linear layer is used to unify the output to the same dimension to obtain the time-dependent features. and frequency-dependent characteristics .

[0060] In order to successfully apply the cross-attention mechanism, we first need to adjust and , reshape both dimensions into The feature fusion method based on the cross attention mechanism has a large requirement for video memory or memory. In order to reduce the requirement for video memory or memory during project deployment and thus be suitable for most application scenarios, this embodiment adjusts the dimension of the feature downward through two linear projection layers to obtain and , whose dimensions are ; Then through the cross attention module The time domain dependency features and frequency domain dependency features are combined to obtain the feature , whose dimensions are ,feature It contains not only time-dependent information, but also frequency-dependent information; Finally, in order to apply the residual connection operation, the feature Through a linear projection layer , reprojecting its dimensions to their original size, and then reshaping its dimensions to match the features Same Dimension , and then with the input Perform residual connection operation to obtain the final output fusion features .

[0061] In addition, in the data input Before the module, in order to alleviate the gradient vanishing problem and reduce the difficulty of optimizing the network model, this embodiment inserts a densely connected module, which contains There are densely connected convolutional blocks, and the dilation factor of each convolutional block is set to .

[0062] 4. Encoder .

[0063] In this embodiment, The structure of the module has been improved in two aspects: (1) a grouped multi-head self-attention mechanism is used to reduce the amount of calculation and speed up the reasoning. (2) progressive downsampling is introduced into the encoder to improve the efficiency of the model.

[0064] 4-1. Progressive downsampling.

[0065] exist In the module, the input sampling rate is changed from Reduce to , this sampling rate remains constant throughout the network, and all attention and convolution operations are performed on a constant time scale. However, the complexity of the attention operation is quadratic relative to the length of the input sequence, which leads to the low efficiency of the model itself. Therefore, this embodiment innovatively reduces this additional overhead by introducing a progressive downsampling mechanism to balance the computational complexity of the entire module.

[0066] like Figure 7 As shown, the 5th Before the module, keep the sampling rate at , and then downsample each input sequence to The downsampling layer is used to merge the redundant information between adjacent embeddings, reducing the attention complexity to times, while also reducing feature redundancy; indivual The module is then followed by a second downsampling to keep the sampling rate constant at , reducing the attention complexity again times, while also reducing feature redundancy.

[0067] 4-2. Grouped multi-head self-attention.

[0068] Considering that the computational complexity of the multi-head self-attention module is quadratic in the sequence length, this will introduce computational asymmetry into the progressive downsampling network model. This is because the first indivual The sequence length of the input data of the module is the longest among all modules, so the efficiency of the model is mainly limited by the previous indivual module.

[0069] In order to solve the above problems, this embodiment innovatively designs a novel attention calculation method, namely, grouped multi-head self-attention. Figure 8As shown, before calculating the conventional multi-head self-attention, grouped multi-head self-attention is applied to group the sequence data along the time dimension. The attention heads in each group share the same parameters. Therefore, each group only needs to undergo one linear transformation, and then the obtained attention matrix is ​​converted into They are sent to their respective group attention heads to calculate the attention, thereby reducing the attention complexity from Down to ,in, is the sequence length, is the feature dimension, is the number of groups; the attention matrix The dimension from × becomes × ,in , Afterwards, the data dimension is reshaped into × , which is convenient for subsequent module calculations.

[0070] The feasibility of grouping multi-head self-attention is that the features of temporally adjacent data have similar features, so the data can be grouped along the time dimension to approximate the conventional multi-head self-attention. indivual Module application grouping self-attention, follow-up The module applies conventional multi-head self-attention to mitigate the efficiency issues caused by computational asymmetry.

[0071] 4-3. Activation function.

[0072] Most blocks in the module use activation function, but the convolution module uses Activation functions, such a heterogeneous design is too complicated and unnecessary. From a practical point of view, using multiple activation functions will complicate hardware deployment. For example, on some low-end edge devices, supporting multiple activation functions will require additional lookup tables or advanced algorithms. To solve this problem, this embodiment uses Activation function replacement Activation function, unify the choice of activation function in the entire model, such as Fig. 9 As shown, the model architecture is simplified.

[0073] 5. Decoder.

[0074] Considering the inference speed, this embodiment adopts a decoding method based on CTC (Connectionist Temporal Classification). Therefore, the decoder is a fully connected layer that maps the features extracted by the encoder to the character probability distribution. superior.

[0075] In this embodiment, a beam search method combined with a language model is used during decoding, and the specific steps are as follows: (1) Initialization: According to the character probability distribution Initialize it to get the size of which is the bundle width, and the bundle contains the most likely candidate word sequence.

[0076] (2) Generate candidate sequences: For each candidate word sequence in the current bundle, use the language model to generate the possible next word based on the probability state of the current time step and the partial word sequence that has been generated.

[0077] (3) Calculate the score: For each generated candidate word sequence, calculate its score. The score usually consists of two parts: the probability generated by the model and the previous cumulative score. The probability generated by the model is provided by the language model.

[0078] (4) Selection Candidate: Select the highest scoring candidate from all generated candidate word sequences sequences as candidate sets for the next step.

[0079] Repeat steps (2) to (4) until words for all time steps are generated.

[0080] 6. Pre-training-fine-tuning.

[0081] Since it is very expensive to obtain a labeled test audio dataset, and the oral proficiency of candidates varies greatly, it is difficult to train a model with good recognition effect simply using the test dataset. To overcome this problem, this embodiment adopts the pre-training-fine-tuning paradigm, and the specific steps are as follows: (1) Use Pre-training on public datasets Model.

[0082] (2) Use Fine-tuning on the internal exam dataset Some parameters of the model.

[0083] For the selection of parameters that need to be fine-tuned, this embodiment fine-tunes the parameters of the time-frequency dependency calculation module, the first 8 The parameters of the module and the parameters of the fully connected layer in the decoder. This is because in the speech recognition model, the front part of the network mainly learns the feature representation of speech and extracts the high-dimensional hidden features of the speech signal. Therefore, this part mainly learns the speech knowledge of the data; while the back part mainly learns the feature representation of the language, which facilitates the decoder to map the features to the character probability distribution. Therefore, this part mainly learns the language knowledge of the data. Based on this, this embodiment mainly fine-tunes the parameters of the front part of the network and fixes the parameters of the back part to achieve better recognition effect on the internal data set.

[0084] In summary, this embodiment provides a spoken English recognition method that can be used in an examination environment, which has the following characteristics and advantages: (1) Innovative design Module stacking to strengthen The original downsampling module in the module is named , in order to better utilize the time-frequency characteristics of speech signals in the downsampling stage and extract richer high-dimensional features.

[0085] (2) Innovative design of time-frequency dependence calculation module , cleverly through dimensional transformation, parallel calculation of the dependencies of the time dimension and the frequency dimension, and then the time-dependent features and the frequency-dependent features are fused through the cross-attention module, and then the densely connected The module is used to solve the gradient vanishing problem and pass more time-frequency domain information in the speech recognition model.

[0086] (3) Innovatively combines progressive downsampling and grouped multi-head self-attention mechanisms. The module adopts a grouped multi-head self-attention mechanism to reduce the attention complexity from Down to ; Later stage The module reduces the data length by half, thereby improving the encoder computational efficiency and making it possible to deploy speech recognition models on edge devices.

[0087] (4) Unification The activation function of the convolution module in the module is used uniformly The activation function simplifies the speech recognition model without reducing the performance, making the deployment of the speech recognition model easier.

[0088] (5) The pre-training-fine-tuning paradigm was adopted to solve the problem of the high cost of obtaining internal datasets. When the internal dataset was severely insufficient, the speech recognition model was first trained using a public dataset with sufficient data volume, and then the internal dataset was used to fine-tune the parameters of the front part of the speech recognition model, which also achieved good recognition performance.

[0089] It should be noted that all data is obtained in compliance with laws and regulations and user consent, and the data is used legally.

[0090] Example 2 This embodiment provides an English spoken language recognition system, including: A feature extraction and downsampling module is configured to obtain a speech signal and extract audio features, and extract time features and frequency features from the audio features; The time-frequency dependency calculation module is configured to calculate the time dependency of the time feature after the dimension transformation, and perform residual connection with the time feature after the dimension transformation to obtain the time dependency feature, calculate the frequency dependency of the frequency feature after the dimension transformation, and perform residual connection with the frequency feature after the dimension transformation to obtain the frequency dependency feature, and fuse the time dependency feature and the frequency dependency feature by adopting the cross attention mechanism and residual connection to obtain the fused feature; The encoding module is configured to use a three-stage encoder to perform progressive downsampling on the fused features; in the first stage, the fused features are grouped along the time dimension using a grouped multi-head self-attention mechanism, and attention is calculated separately for each group; The decoding module is configured to decode the output features of the three-stage encoder to obtain character probability distribution, and thereby obtain speech recognition results.

[0091] It should be noted that the above modules correspond to the steps described in Example 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above Example 1. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer executable instructions.

[0092] In further embodiments, there is also provided: An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in Embodiment 1 is performed. For the sake of brevity, it will not be described in detail here.

[0093] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0094] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0095] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the method described in Example 1 is completed.

[0096] The method in Example 1 can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it is not described in detail here.

[0097] A computer program product includes a computer program, and when the computer program is executed by a processor, the method described in embodiment 1 is implemented.

[0098] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer executable instructions, such as instructions included in a program module, which are executed in a device on a real or virtual processor of the target to perform the process / method as described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functions of program modules can be combined or divided between program modules as needed. Machine executable instructions for program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in local and remote storage media.

[0099] The computer program code for implementing the method of the present invention can be written in one or more programming languages. These computer program codes can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the computer or other programmable data processing device, causes the function / operation specified in the flow chart and / or block diagram to be implemented. The program code can be executed completely on a computer, partially on a computer, as an independent software package, partially on a computer and partially on a remote computer or completely on a remote computer or server.

[0100] In the context of the present invention, computer program codes or related data may be carried by any appropriate carrier to enable a device, apparatus or processor to perform the various processes and operations described above. Examples of carriers include signals, computer readable media, etc. Examples of signals may include electrical, optical, radio, acoustic or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0101] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0102] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for recognizing spoken English, characterized in that: include: Acquire speech signals and extract audio features, and extract time features and frequency features from the audio features; After the time feature is transformed in dimension, the time dependency is calculated, and the time dependency feature is obtained by residual connection with the transformed time feature. After the frequency feature is transformed in dimension, the frequency dependency is calculated, and the frequency dependency feature is obtained by residual connection with the transformed frequency feature. The time dependency feature and the frequency dependency feature are fused by cross attention mechanism and residual connection to obtain fused feature. The fused features are progressively downsampled using a three-stage encoder. In the first stage, the fused features are grouped along the time dimension using a grouped multi-head self-attention mechanism, and attention is calculated separately for each group. The output features of the three-stage encoder are decoded to obtain the character probability distribution, and the speech recognition result is obtained based on it.

2. A method for recognizing spoken English as claimed in claim 1, characterized in that: The process of extracting time features and frequency features from audio features includes: using stacked The modules constitute a downsampling module, and The channels of the module are increased successively to extract time features and frequency features; specifically: The audio features are processed by stride The convolutional layer performs downsampling in the time dimension and frequency dimension, and then inputs into the first In the module; First The output of the module is sequentially passed through the After the convolution layer and batch normalization operation, the input is In the module; second The output of the module is sequentially passed through the After the convolution layer and batch normalization operation, the input is In the module; third The output of the module is transformed in dimension through layer normalization and linear layer operations to extract time and frequency features.

3. The method for recognizing spoken English as claimed in claim 1, characterized in that: The process of extracting time-dependent features and frequency-dependent features includes: reshaping the time features and frequency features respectively, and then using the time domain Block and frequency domain The blocks calculate the time domain dependency and frequency domain dependency respectively, and then unify them to the same dimension through the linear layer after residual connection, so as to obtain the time dependency feature and frequency dependency feature respectively.

4. The method for recognizing spoken English as claimed in claim 1, characterized in that: The feature fusion process includes: fusing the time-domain dependent features and the frequency-domain dependent features through the cross-attention mechanism, and then performing residual connection with the input time features and frequency features after linear projection and dimensionality reshaping to obtain the fused features.

5. The method for recognizing spoken English as claimed in claim 1, characterized in that: The grouped multi-head self-attention mechanism includes: using the grouped multi-head self-attention mechanism to group along the time dimension, the attention heads in each group share the same parameters, and after each group is linearly transformed, the obtained attention matrix is ​​sent to the attention heads in each group to calculate the attention, thereby reducing the attention complexity from Down to ,in, is the sequence length, is the feature dimension, is the number of groups, and the dimension of the attention matrix changes from × becomes × , , .

6. A method for recognizing spoken English as claimed in claim 1, characterized in that: The method for recognizing spoken English also includes: the speech recognition result is obtained by processing the trained speech recognition module, and the speech recognition module is trained using a pre-training-fine-tuning paradigm, wherein the parameters of the time-frequency dependency calculation process, the parameters in the first stage encoder and the parameters of the decoding process are fine-tuned, and the uniform use of Activation function.

7. An English spoken language recognition system, characterized in that: include: A feature extraction and downsampling module is configured to obtain a speech signal and extract audio features, and extract time features and frequency features from the audio features; The time-frequency dependency calculation module is configured to calculate the time dependency of the time feature after the dimension transformation, and perform residual connection with the time feature after the dimension transformation to obtain the time dependency feature, calculate the frequency dependency of the frequency feature after the dimension transformation, and perform residual connection with the frequency feature after the dimension transformation to obtain the frequency dependency feature, and fuse the time dependency feature and the frequency dependency feature by adopting the cross attention mechanism and residual connection to obtain the fused feature; The encoding module is configured to use a three-stage encoder to perform progressive downsampling on the fused features; in the first stage, the fused features are grouped along the time dimension using a grouped multi-head self-attention mechanism, and attention is calculated separately for each group; The decoding module is configured to decode the output features of the three-stage encoder to obtain character probability distribution, and thereby obtain speech recognition results.

8. An electronic device, characterized in that: The method comprises a memory and a processor and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method according to any one of claims 1 to 6 is completed.

9. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the method described in any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images