An English spoken language recognition method, system, device, medium and program product

Abstract

The application discloses an English oral recognition method, system, device, medium and program product, relates to the technical field of speech recognition, and comprises the following steps: extracting time characteristics and frequency characteristics of audio characteristics; performing dimension transformation on the time characteristics and the frequency characteristics, and calculating the dependency of the time dimension and the frequency dimension in parallel, fusing time-dependent characteristics and frequency-dependent characteristics through a cross-attention module, adopting a three-stage encoder to perform progressive downsampling processing, adopting a grouping multi-head self-attention mechanism in the first stage to perform grouping calculation along the time dimension, finally obtaining a speech recognition result through decoding, paying attention to the time-frequency characteristics of speech characteristics, and improving the calculation efficiency of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, in particular to an English spoken language recognition method, system, device, medium and program product. BACKGROUND

[0002] Speech recognition is a technology that converts speech signals into text or commands, also known as automatic speech recognition (ASR). Speech recognition systems can convert the spoken speech of a speaker into written text, allowing computers to understand and process speech input.

[0003] Before the rise of deep learning technology, speech recognition mainly used traditional signal processing and pattern matching techniques. Pattern matching techniques include dynamic time warping and hidden Markov models, which are used to model and match the features of speech signals to recognize the speech of a speaker.

[0004] However, traditional speech recognition methods have some significant drawbacks. First, such methods are usually based on limited vocabularies and hand-designed grammar rules, limiting their ability to handle a wide range of vocabularies and more free language expressions. Second, such methods rely on hand-designed feature extraction methods, such as Mel-frequency cepstral coefficients and linear predictive coding, which require domain expertise and may not fully capture abstract features in speech signals. Finally, the models used by such methods, such as Gaussian mixture models and hidden Markov models, have certain limitations in capturing complex speech patterns and long-term dependencies.

[0005] Deep learning models can better handle complex speech patterns, larger-scale datasets, and are more robust to environmental noise. However, despite the significant results achieved by deep learning-based speech recognition methods, there are still some shortcomings, such as the model's reliance on time dimension dependency calculations for audio data, and insufficient data leading to poor actual recognition results.

[0006] For example, a novel convolutional enhancement architecture in the field of speech recognition . Because it can simultaneously capture global and local features from audio signals, the module has become not only a common model for tasks, but also a model for various end-to-end speech processing tasks. However, most of its model functions are assigned to time modeling, specifically time dimension dependency calculations. While speech is a time series signal, unlike text data, the features of speech signals are in the time-frequency domain, so the frequency domain also plays an indispensable role.

[0007] In addition, the main driving force of deep neural networks is the availability of large amounts of training data, but this is not always feasible for speech recognition. This is because the cost of collecting large-scale audio data containing text labels is high, especially in the oral test scene, most of the examinees' oral pronunciation is not standard and unified, and a large part of the audio contains a lot of silence or intermittent fragments, which greatly increases the cost of transcription. It is difficult to train a high-accuracy speech recognition model from such low-quality audio data, which affects the recognition accuracy. SUMMARY

[0008] To solve the above problems, the present application provides an English oral recognition method, system, device, medium and program product, which designs Stacked reinforcement of Convolutional Network Net Generation (Convolutional Network Net Generation, next generation convolutional neural network) module The original down-sampling module in the module focuses on the time-frequency characteristics of the speech features, a time-frequency dependency calculation module is designed after the down-sampling module to calculate the dependency of the time dimension and the frequency dimension at the same time, and the two kinds of features are fused; at the same time, the gradual down-sampling and the grouped multi-head self-attention mechanism are fused in the encoder to improve the calculation efficiency of the model.

[0009] To achieve the above purpose, the present application adopts the following technical solutions:

[0010] In a first aspect, the present application provides an English oral recognition method, comprising:

[0011] Obtaining a speech signal and extracting audio features, extracting time features and frequency features from the audio features;

[0012] Calculating time dependency after dimension transformation of the time features, and performing residual connection with the dimension-transformed time features to obtain time dependency features, calculating frequency dependency after dimension transformation of the frequency features, and performing residual connection with the dimension-transformed frequency features to obtain frequency dependency features, and adopting cross-attention mechanism fusion and residual connection of the time dependency features and the frequency dependency features to obtain fused features;

[0013] Performing gradual down-sampling processing on the fused features by a three-stage encoder; in the first stage, the grouped multi-head self-attention mechanism is used to group along the time dimension, and attention is calculated separately for each group;

[0014] Decoding the output features of the three-stage encoder to obtain character probability distribution, and obtaining speech recognition results therefrom.

[0015] As an optional implementation, the process of extracting time features and frequency features from audio features includes: adopting a stacked The module constitutes a down-sampling module, and The channels of the module are sequentially increased to extract the time feature and the frequency feature; specifically:

[0016] The audio feature is down-sampled on the time dimension and the frequency dimension through a convolution layer with a stride of , and then is input into the first module after a batch normalization operation.

[0017] The output of the first module is input into the second module after sequentially passing through a convolution layer with a stride of and a batch normalization operation.

[0018] The output of the second module is input into the third module after sequentially passing through a convolution layer with a stride of and a batch normalization operation.

[0019] The output of the third module is dimensionally converted after sequentially passing through a layer normalization operation and a linear layer operation to complete the extraction of the time feature and the frequency feature.

[0020] As an optional implementation, the process of extracting the time-dependent feature and the frequency-dependent feature includes: after the time feature and the frequency feature are respectively subjected to a dimension remodeling operation, a time-domain block and a frequency-domain block are used to respectively calculate the time-dependent and the frequency-dependent, and then the residual connection is performed to be unified to the same dimension through a linear layer, so as to respectively obtain the time-dependent feature and the frequency-dependent feature.

[0021] As an optional implementation, the process of feature fusion includes: the time-dependent feature and the frequency-dependent feature are fused through a cross-attention mechanism, and then are subjected to linear projection and dimension remodeling before being subjected to residual connection with the input time feature and the frequency feature to obtain the fused feature.

[0022] As an optional implementation, the grouped multi-head self-attention mechanism includes: the grouped multi-head self-attention mechanism is used to group along the time dimension, the attention heads in each group share the same parameters, and each group is subjected to a linear transformation to send the obtained attention matrix into the respective group attention head to calculate the attention, so that the attention complexity is reduced from to , wherein, is the sequence length, is the feature dimension, is the number of groups, and the dimension of the attention matrix is changed from × to X , , .

[0023] As an alternative embodiment, the English spoken language recognition method further comprises: the speech recognition result is obtained by processing a trained speech recognition module, and the speech recognition module is trained by using a pre-training-fine-tuning paradigm, wherein the parameters of the time-frequency dependency calculation process, the parameters in the first stage encoder and the parameters of the decoding process are fine-tuned, and the activation function is uniformly used.

[0024] In a second aspect, the present application provides an English spoken language recognition system, comprising:

[0025] a feature extraction and down-sampling module configured to obtain a speech signal and extract audio features, and extract time features and frequency features from the audio features;

[0026] a time-frequency dependency calculation module configured to calculate time dependency after dimension transformation of the time features, and perform residual connection with the dimension-transformed time features to obtain time dependency features, calculate frequency dependency after dimension transformation of the frequency features, and perform residual connection with the dimension-transformed frequency features to obtain frequency dependency features, and obtain fusion features after fusion and residual connection of the time dependency features and the frequency dependency features by using a cross-attention mechanism;

[0027] an encoding module configured to perform progressive down-sampling processing on the fusion features by using a three-stage encoder; wherein in the first stage, the fusion features are grouped and attention is calculated for each group along the time dimension by using a grouped multi-head self-attention mechanism;

[0028] a decoding module configured to decode the output features of the three-stage encoder to obtain character probability distribution, and obtain a speech recognition result based on the character probability distribution.

[0029] In a third aspect, the present application provides an electronic device, comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein when the computer instructions are run by the processor, the method of the first aspect is completed.

[0030] In a fourth aspect, the present application provides a computer-readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, the method of the first aspect is completed.

[0031] In a fifth aspect, the present application provides a computer program product comprising a computer program, wherein when the computer program is executed by a processor, the method of the first aspect is completed.

[0032] Compared with the prior art, the present application has the following beneficial effects:​

[0033] The application provides an English oral recognition method which can be used in a test environment.

[0034] (1) The application innovatively designs a stacked to strengthen the original down-sampling module in the module to better utilize the time-frequency characteristics of the speech signal in the down-sampling stage and extract more abundant high-dimensional features.

[0035] (2) The application innovatively designs a time-frequency dependency calculation module, which ingeniously calculates the dependency of the time dimension and the frequency dimension through dimension transformation in parallel, and then fuses the time-dependent features and the frequency-dependent features through a cross-attention module, and then introduces a (Dense Convolutional Network, Densely Connected Convolutional Network) module to solve the gradient disappearance problem and pass more time-frequency domain information in the speech recognition model.

[0036] (3) The application innovatively fuses progressive down-sampling and grouped multi-head self-attention mechanism. The encoder in the first stage adopts the grouped multi-head self-attention mechanism, which reduces the attention complexity from to ; the encoder in the later stage halves the data length, thereby improving the calculation efficiency of the encoder and making it possible to deploy the speech recognition model on edge devices.

[0037] (4) The application unifies the activation function of the convolution module in the encoder and uniformly uses activation function, which ensures that the performance does not decrease while simplifying the speech recognition model, making the deployment of the speech recognition model more simple.

[0038] (5) The application adopts the pre-training-fine-tuning paradigm to solve the problem of high cost of obtaining internal data sets. In the case of serious lack of internal data sets, the speech recognition model is first trained using a public data set with sufficient data, and then the parameters of the front part of the speech recognition model are fine-tuned using the internal data set, which also achieves good recognition performance.

[0039] The advantages of the additional aspects of the application will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0040] The drawings accompanying the specification of the application form a part of the application and serve to provide further understanding of the application, the illustrative embodiments of the application and their description serve to explain the application without constituting an improper limitation of the application.

[0041] Figure 1 For Structure diagram of the module;

[0042] Figure 2 For Structure diagram of the convolution module in the module;

[0043] Figure 3 Overall structure diagram of the speech recognition model provided for Embodiment 1 of the present application;

[0044] Figure 4 Flowchart of the audio feature extraction provided for Embodiment 1 of the present application;

[0045] Figure 5 Structure diagram of the down-sampling module provided for Embodiment 1 of the present application;

[0046] Figure 6 Structure diagram of the time-frequency dependency calculation module provided for Embodiment 1 of the present application;

[0047] Figure 7 Progressive down-sampling schematic diagram provided for Embodiment 1 of the present application;

[0048] Figure 8 Group multi-head self-attention schematic diagram provided for Embodiment 1 of the present application;

[0049] Figure 9 Structure diagram of the convolution module in the speech recognition model provided for Embodiment 1 of the present application. DETAILED DESCRIPTION

[0050] The present application will be further described below in conjunction with the accompanying drawings and embodiments.

[0051] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as would be understood by one of ordinary skill in the art to which the present application pertains.

[0052] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should be further understood that the terms "comprise" and "include" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a list of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to such processes, methods, products, or devices.

[0053] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0054] Example 1

[0055] Currently, a novel convolutional enhancement method has been proposed in the field of speech recognition. Architecture Because it can simultaneously capture global and local features from audio signals, it enables... Modules not only become It is a commonly used model for tasks and has also become a model for various end-to-end speech processing tasks. Each The structure of the module is as follows Figure 1 As shown, each The module contains two fully connected layers. In two Between them, a multi-head self-attention module with four heads is used. This is followed by a convolutional module. .

[0056] Its convolutional module structure is as follows Figure 2 As shown, the layers are, in order: layer normalization, pointwise convolutional layer, and... An activation function is used to reduce the vanishing gradient problem. Then, the input is fed into a function with... A one-dimensional depthwise convolutional layer with activation function is then fed into another pointwise convolutional layer. Finally, it is used... The layer performs regularization on the network.

[0057] Specifically, for input features each The module's calculation process is as follows:

[0058] (1);

[0059] (2);

[0060] (3);

[0061] (4);

[0062] in, Represents a fully connected layer, whose output is multiplied by After input Adding them together yields intermediate features. ; Represents a multi-head self-attention layer, whose output and input... Adding them together yields intermediate features. ; Represents a convolutional layer, its output and input. Adding them together yields intermediate features. ;at last, After a fully connected layer Multiply by Then input The sums are then processed through a normalization layer. Get output .

[0063] Although deep learning-based speech recognition methods have achieved remarkable results, they still have some shortcomings, such as the model's reliance on the temporal dimension of audio data for computation, and insufficient data leading to poor actual recognition performance; as mentioned above. The module allocates most of its modeling functionality to time modeling, specifically in the calculation of time-dependent dependencies. Although speech is a time-series signal, unlike text data, speech signals are characterized in the time-frequency domain, thus the frequency domain also plays another indispensable role.

[0064] Therefore, in order to pay more attention to the time-frequency characteristics of speech features and make the model more efficient and accurate, this embodiment proposes a novel speech recognition model; firstly, it designs a stacking of ConvNext (Convolutional Network NetGeneration) modules to enhance performance. The module originally contained a downsampling module. Then, following the downsampling module, a time-frequency dependency calculation module was designed to simultaneously calculate the dependencies in both the time and frequency dimensions, fuse the two features, and then input them into a densely connected module. The Densely Connected Convolutional Networks (DCNN) module addresses the vanishing gradient problem while also transmitting more time-frequency domain information within the network.

[0065] The improvements to the downsampling module and the introduction of the time-frequency dependence calculation module inevitably increased the computational load, leading to model efficiency issues. To address this problem, firstly, improvements were made at the macroscopic level. The overall architecture of the module introduces progressive downsampling. Within the module; simultaneously, microscopic improvements. The module's computation method employs a grouped multi-head self-attention mechanism, while simultaneously unifying the selection of activation functions to further improve model computational efficiency.

[0066] Furthermore, a major driving force behind powerful deep neural networks is the availability of large amounts of training data, which is not always feasible for speech recognition. This is because collecting large-scale audio data with text labels is very costly, especially in oral exam scenarios, where most test takers' pronunciation is not standard and inconsistent, and the audio contains a large portion of silence or fragmented segments, which significantly increases the cost of transcription. Such low-quality audio data makes it difficult to train high-accuracy speech recognition models, causing numerous inconveniences for further work after recognition.

[0067] To address this issue, this embodiment employs a pre-training-fine-tuning paradigm to train the speech recognition model. Specifically, the speech recognition model is first pre-trained on a public dataset to enable it to generalize effectively and extract useful features. Then, some parameters of the speech recognition model are fine-tuned on a small number of internal exam datasets, allowing the model to achieve a good recognition accuracy even with limited internal exam datasets. This overcomes the limitation of obtaining large amounts of transcribed internal exam datasets.

[0068] like Figure 3 The diagram shows the overall structure of the speech recognition model. Its input is audio features (Mel-Cepstral Coefficients). First, the downsampling module is used to reduce the sequence length of the data, improve the computational efficiency of subsequent modules, and increase the model's decoding speed. The time-frequency dependency calculation module is used to extract the time and frequency dependencies of the data in parallel and fuse them to obtain new features. The encoder is used to extract high-dimensional hidden features of the audio, which are used as input to the decoder. The decoder identifies the corresponding text sequence based on the high-dimensional hidden features output by the encoder.

[0069] Each module will be described in detail below.

[0070] 1. Feature extraction module.

[0071] Mel cepstral coefficients ( MFCCs are one of the commonly used feature extraction methods in speech signal processing and speech recognition. In this embodiment, MFCCs are used as input features of the speech recognition model.

[0072] Specifically, such as Figure 4 As shown:

[0073] (1) Apply a pre-emphasis filter to the speech signal to amplify the high frequency, thereby improving the signal-to-noise ratio of the speech signal, eliminating the effects of the vocal cords and lips during the speech process, and avoiding numerical problems during the short-time Fourier transform operation.

[0074] (2) After pre-emphasis, the speech signal is divided into short-time frames by framing and windowing. Because in most cases, the speech signal is non-stationary, it is meaningless to perform a Fourier transform on the entire speech signal, as the frequency profile of the speech signal will be lost over time. The speech signal is a short-time stationary signal, so it is necessary to perform a Fourier transform on the short-time frames and obtain a good approximation of the signal frequency profile by connecting adjacent frames.

[0075] In this embodiment, the speech signal is framed as frame, The frame length of the speech signal is Each sampling point; simultaneously, the frame shift is set to [number] sampling points; After segmenting the speech signal into frames, each frame is multiplied by a window function to increase the continuity at the left and right ends of the frame, counteracting the assumption of the Fast Fourier Transform (FFT) that the data is infinite, and reducing spectral leakage.

[0076] The form of the Hanming window is as follows:

[0077] (5);

[0078] in, , It is the window length. It is a parameter that can be set manually.

[0079] (3) Since it is usually difficult to see the characteristics of a speech signal from its transformation in the time domain, it is usually done by... The transformation converts the signal to an energy distribution in the frequency domain for observation. Different energy distributions represent different speech characteristics. Next, each frame of the windowed signal is analyzed... Point FFT is used to calculate the spectrum, also known as short-time Fourier transform (FFT). Then, the power spectrum is calculated, and the spectral energy P of the speech signal is obtained by taking the square of the modulus of the speech signal's spectrum.

[0080] (6);

[0081] in, It is the first The audio signal consists of frames, where N is the total number of frames.

[0082] (4) Pass the power spectrum through Filter bank ( Filter bank for taking To extract the frequency band, use a triangular filter (one filter per filter). The input features are of dimension ; the final feature dimension is . , represent the number of samples in the training set ), represent the time dimension, represent the frequency dimension.

[0083] 2, a down-sampling module .

[0084] In the down-sampling module, a module is introduced to better extract the time dimension feature and the frequency dimension feature. ConvNext is a kind of convolutional neural network (CNN) structure similar to deep residual network (ResNet), compared with , has a larger kernel size and depth convolution, making it more accurate and efficient. The kernel size of the embodiment is set to 7. As shown in (a) of

[0085] , the down-sampling module Figure 5 is a stack of modules with down-sampling layers, a total of three stages are designed, and the channel size of each stage is gradually increased to . For an input audio feature with a frame rate of , the processing process is as follows:

[0086] (7);

[0087] (8);

[0088] (9);

[0089] (10);

[0090] wherein, represents a layer normalization operation; represents a linear layer operation; , , represent the operations of the first convolutional layer, the second convolutional layer and the third convolutional layer; represents a batch normalization operation; , , represent the operations of the stacked first module, the second module and the third module.

[0091] Specifically, for an input ​​, the first stage (formula (7)), first through the convolution layer with a stride of down-sampling in the time dimension and the frequency dimension to reduce the computational complexity of the subsequent steps, and then using batch normalization to stabilize the training, and then input into the module to obtain the output ; the second stage (formula (8)) and the third stage (formula (9)) do not perform down-sampling operations, and the strides of the convolution layers and are set to , because the down-sampling here will damage the results, so the subsequent time down-sampling is placed in the encoder , and the sampling rate is . Finally, the data dimension is converted to the input dimension of the subsequent time-frequency dependent calculation module using the linear layer to obtain the output feature . Assuming that the dimension of the input audio feature is , after the module, the dimension of the output feature becomes .

[0092] Through the above design, the down-sampling module can simultaneously extract the time features and frequency features in the audio features, thereby fully utilizing the time-frequency characteristics of the speech signal.

[0093] As shown in (b) in Figure 5 , the module includes a deep convolution, two point convolutions, a normalization layer, an activation layer, and a scaling layer; wherein the deep convolution is two-dimensional because the speech features have a time-frequency dimension, and the introduction of the deep convolution in this embodiment makes the calculation more efficient; the two point-by-point convolutions have a similar effect to the feedforward sub-module in the network. .

[0094] Specifically, taking the first module as an example, for the input feature , the processing process is as follows:

[0095] (11);

[0096] (12);

[0097] (13);

[0098] wherein represents the operation of the depth separable convolution layer.​ This indicates the first reshaping operation, used to reduce the audio feature dimension from... Remodeling , represent , Represents the time dimension. Represents the frequency dimension. Represents the channel dimension; The representation layer normalization operation, the output of the first reshaping operation is passed through a layer normalization layer to obtain the first output. ; This indicates the first point of convolution operation, where the output channel is the same as the input channel. times; This represents the operation of the activation function; the output of the first point of the convolution operation is processed by... The activation function yields the second output. ; This indicates the second-point convolution operation, where the output channel is the same as the input channel. ; This indicates the operation of scaling the layer; This indicates the second reshaping operation, used to change the data dimension from... Remodeling Then, combined with input features Add to get the output .

[0099] 3. Time-frequency dependence calculation module .

[0100] Same downsampling module Similarly, considering the time-frequency characteristics of audio signals, the encoder should simultaneously calculate the time dependence and frequency dependence of the data. Therefore, this embodiment innovatively uses... The module features a novel time-frequency dependency calculation module. By using the self-attention module in the Conformer block, features with time-dependent dimensions can be computed separately. Features dependent on frequency dimension ,like Figure 6 As shown, using the time domain Block and frequency domain Blocks are used to capture time-domain dependencies and frequency-domain dependencies respectively, time domain Block and frequency domain The block structure adopts Figure 1 The structure shown.

[0101] Specifically, given a set of data that simultaneously contains both temporal and frequency features... Its dimensions are , The module's calculation process is as follows:

[0102] (14);

[0103] (15);

[0104] (16);

[0105] (17);

[0106] in, This indicates a time-feature dimension reshaping operation; This indicates a frequency feature dimension reshaping operation; This indicates the operation of the time-domain module; This indicates the operation of the frequency domain module; Operations representing linear projection layers; This indicates a cross-attention feature fusion operation; This indicates the operation of reshaping the fusion feature dimensions.

[0107] Input The dimensions are respectively through and Reshaped into and Then they are processed separately through the time domain module. and frequency domain module Features containing time dependence and frequency dependence are calculated separately. The input and output are then joined using a residual connection, and finally, a linear layer is applied to unify the output to the same dimension, yielding the time-dependent features. and frequency-dependent features .

[0108] In order to successfully apply the cross-attention mechanism, it is first necessary to adjust... and The dimensions of both will be reshaped into Feature fusion methods based on cross-attention mechanisms have high requirements for GPU or RAM. To reduce the GPU or RAM requirements during project deployment and thus suit most application scenarios, this embodiment uses two linear projection layers to adjust the dimension of the features downwards, obtaining the following results: and All of their dimensions are ;

[0109] Then through the cross-attention module By fusing time-domain dependency features and frequency-domain dependency features, the following features are obtained. Its dimensions are ,feature Not only contains time-dependent information, but also contains frequency-dependent information;

[0110] Finally, in order to apply the residual connection operation, the feature is projected through a linear projection layer to the original size, and then the dimension is reshaped to the same dimension as the feature , and then a residual connection operation is performed with the input to obtain the final output fusion feature .

[0111] In addition, before the data input module, in order to alleviate the gradient vanishing problem and reduce the optimization difficulty of the network model, the embodiment inserts a densely connected module, which contains convolutional blocks with dense connection, and the expansion factor of each convolutional block is set to .

[0112] 4、Encoder .

[0113] In this embodiment, the structure of the module is improved, mainly in two aspects: (1) adopting a grouped multi-head self-attention mechanism to reduce the computational complexity and speed up the inference; (2) introducing progressive down-sampling into the encoder to improve the model efficiency.

[0114] 4-1、Progressive down-sampling.

[0115] In the module, the input sampling rate is reduced from to through the convolutional subsampling block at the bottom of the network, and this sampling rate remains constant throughout the network, and all attention and convolution operations are performed on a constant time scale. However, the complexity of the attention operation is quadratic with respect to the length of the input sequence, which results in low efficiency of the model itself. Therefore, the embodiment innovatively introduces a progressive down-sampling mechanism to reduce this additional overhead and balance the computational load of the entire module.

[0116] As shown in Figure 7 , the sampling rate is kept at before the 5th module, and then the sampling rate of each input sequence is reduced to by using a down-sampling layer, which is used to merge the redundant information between adjacent embeddings and reduce the attention complexity by times, while also reducing the redundancy of the features; the first After the module, a second downsampling is performed to maintain the sampling rate. This further reduces the attention complexity. This doubles the efficiency and also reduces feature redundancy.

[0117] 4-2. Grouped multi-head self-attention.

[0118] Given that the computational complexity of a multi-head self-attention module is the square of the sequence length, this introduces computational asymmetry into progressive downsampling network models. This is because the encoder's first... indivual The input data sequence length of this module is the longest among all modules, thus the model efficiency is mainly limited by the first few modules. indivual Module.

[0119] To address the aforementioned issues, this embodiment innovatively designs a novel attention calculation method: grouped multi-head self-attention. For example... Figure 8 As shown, before computing conventional multi-head self-attention, grouped multi-head self-attention is applied to group the sequence data along the time dimension. The attention heads within each group share the same parameters. Thus, each group only needs to undergo one linear transformation, and then the resulting attention matrix is ​​calculated. Each input is fed into its respective attention head for attention calculation, thereby reducing the attention complexity from... Down to ,in, It is the sequence length. It is the feature dimension. It represents the number of groups; attention matrix. Dimensions from × Become × ,in , Subsequently, the data dimensions were reshaped into... × This facilitates calculations in subsequent modules.

[0120] The feasibility of grouped multi-head self-attention is based on the fact that temporally adjacent data have similar characteristics. Therefore, data can be grouped along the time dimension to approximate conventional multi-head self-attention. In this embodiment, before the encoder... indivual The module applies grouped self-attention, and the following... The module employs conventional multi-head self-attention, thereby mitigating the efficiency issues caused by computational asymmetry.

[0121] 4-3. Activation function.

[0122] Most blocks in the module use activation functions, but the convolution module uses activation functions. Such a heterogeneous design is too complex and unnecessary. From a practical point of view, using multiple activation functions will complicate hardware deployment. For example, on some low-end edge devices, supporting multiple activation functions will require additional lookup tables or advanced algorithms. To solve this problem, the present embodiment replaces activation functions with activation functions, unifying the selection of activation functions throughout the model, as Figure 9 shown, thereby simplifying the model architecture.

[0123] 5. Decoder.

[0124] Considering the inference speed, the present embodiment adopts a decoding method based on CTC (Connectionist Temporal Classification), so the decoder is a fully connected layer that maps the features extracted by the encoder to a character probability distribution .

[0125] In the present embodiment, a beam search method combined with a language model is used during decoding, and the specific steps are as follows:

[0126] (1) Initialization: initialization is performed according to the character probability distribution , obtaining a beam of size , where is the beam width, and the beam contains the most likely candidate word sequences.

[0127] (2) Generating candidate sequences: for each candidate word sequence in the current beam, according to the probability state of the current time step and the generated partial word sequence, use the language model to generate the next possible word.

[0128] (3) Score calculation: for each generated candidate word sequence, calculate its score. The score is usually composed of two parts: the probability of model generation and the previous cumulative score. Among them, the probability of model generation is provided by the language model.

[0129] (4) Selecting candidates: select the top sequences from all generated candidate word sequences as the candidate set for the next step.

[0130] Repeat steps (2) to (4) until all time steps of words are generated.

[0131] 6. Pre-training-fine-tuning.

[0132] Since the cost of obtaining a labeled exam audio dataset is high, and the oral English level of test takers is uneven, it is difficult to train a model with good recognition effect simply using the exam dataset. In order to overcome this difficulty, the embodiment adopts the pre-training-fine-tuning paradigm, and the specific steps are as follows:

[0133] (1) Pre-training the model using the public dataset of .

[0134] (2) Fine-tuning part of the parameters of the model using the internal exam dataset of .

[0135] For the selection of parameters that need to be fine-tuned, the embodiment fine-tunes the parameters of the time-frequency dependency calculation module, the parameters of the first 8 modules in the encoder, and the parameters of the full connection layer in the decoder. This is because in a speech recognition model, the front part of the network mainly learns the feature representation of speech, and extracts the high-dimensional hidden features of the speech signal, so this part mainly learns the speech knowledge of the data; and the rear part mainly learns the feature representation of the language, so as to facilitate the decoder to map the features to the character probability distribution, so this part mainly learns the language knowledge of the data. Based on this, the embodiment mainly fine-tunes the parameters of the front part of the network, and fixes the parameters of the rear part, so as to achieve better recognition effect on the internal dataset.

[0136] In summary, the embodiment provides an English oral English recognition method that can be used in an exam environment, which has the following characteristics and advantages:

[0137] (1) The stacking of the module is innovatively designed to strengthen the original down-sampling module in the module, named , so as to better utilize the time-frequency characteristics of the speech signal in the down-sampling stage, and extract more rich high-dimensional features.

[0138] (2) The time-frequency dependency calculation module is innovatively designed, which skillfully calculates the dependency of the time dimension and the frequency dimension through dimension transformation in parallel, and then fuses the time dependency features and the frequency dependency features through the cross-attention module, and then introduces the module with dense connection to solve the gradient disappearance problem, and at the same time, more time-frequency domain information is transmitted in the speech recognition model.

[0139] (3) The progressive down-sampling and grouped multi-head self-attention mechanism are innovatively fused, and the module in the first stage adopts the grouped multi-head self-attention mechanism, which reduces the attention complexity from to ​​The later stage The module halves the data length, thereby improving the computational efficiency of the encoder and making it possible to deploy the speech recognition model on edge devices.

[0140] (4) The activation function of the convolution module in the module is unified to use The activation function, which ensures that the performance does not decrease while simplifying the speech recognition model, makes the deployment of the speech recognition model more simple.

[0141] (5) The pre-training-fine-tuning paradigm is adopted to solve the problem of high cost of obtaining internal data sets. In the case of serious lack of internal data sets, the speech recognition model is first trained using a public data set with sufficient data, and then the parameters of the front part of the speech recognition model are fine-tuned using the internal data set, which also achieves good recognition performance.

[0142] It should be noted that all data is obtained in accordance with laws and regulations and user consent, and the data is legally applied.

[0143] Embodiment 2

[0144] The embodiment provides an English oral English recognition system, comprising:

[0145] The feature extraction and down-sampling module is configured to obtain a speech signal and extract audio features, and extract time features and frequency features from the audio features;

[0146] The time-frequency dependency calculation module is configured to calculate time dependency after dimension transformation of the time features, and perform residual connection with the dimension-transformed time features to obtain time dependency features, calculate frequency dependency after dimension transformation of the frequency features, and perform residual connection with the dimension-transformed frequency features to obtain frequency dependency features, and after fusion and residual connection of the time dependency features and the frequency dependency features using the cross-attention mechanism, obtain the fusion features;

[0147] The encoding module is configured to perform progressive down-sampling processing on the fusion features using a three-stage encoder; in the first stage, the fusion features are grouped and multi-head self-attention mechanism is used to calculate attention along the time dimension;

[0148] The decoding module is configured to decode the output features of the three-stage encoder to obtain a character probability distribution, and obtain a speech recognition result based on the character probability distribution.

[0149] ​It should be noted that the above modules correspond to the steps described in Embodiment 1, and the above modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules can be executed in a computer system such as a set of computer executable instructions as part of a system.

[0150] In more embodiments, there are also provided:

[0151] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, when executed by the processor, complete the method described in Embodiment 1. For brevity, it will not be described here.

[0152] It should be understood that in the embodiments, the processor can be a central processing unit CPU, and the processor can also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, ready-to-program gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0153] The memory can include read-only memory and random access memory, and provide instructions and data to the processor, and a part of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0154] A computer readable storage medium for storing computer instructions, when executed by a processor, completes the method described in Embodiment 1.

[0155] The method in Embodiment 1 can be directly embodied as a hardware processor to complete, or a combination of hardware and software modules in the processor to complete. The software module can be located in a storage medium mature in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory to complete the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0156] A computer program product includes a computer program, which, when executed by a processor, implements the method described in Embodiment 1.

[0157] The present application also provides at least one computer program product tangibly stored on a non-transitory computer readable storage medium. The computer program product includes computer executable instructions, for example, instructions embodied in program modules, executed by devices at the target real or virtual processor to perform the processes / methods as described above. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules can be combined or split between program modules as desired in various embodiments. Machine executable instructions for program modules can be executed within a local or distributed device. In a distributed device, program modules can be located in local and remote memory storage devices.

[0158] Computer program code for carrying out operations of the present application can be written in one or more programming languages. These computer program codes can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program codes, when executed by the computer or other programmable data processing apparatus, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program codes can be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0159] In the context of the present application, the computer program code or related data can be carried by any suitable carrier to enable the device, apparatus or processor to perform the various processes and operations described above. Examples of the carrier include a signal, a computer readable medium, etc. Examples of the signal can include an electrical, optical, radio, sound or other forms of propagated signals, such as a carrier wave, an infrared signal, etc.

[0160] Those skilled in the art can understand that the units and algorithm steps of the examples described in conjunction with the embodiments can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0161] The above describes the specific embodiments of the present application in conjunction with the accompanying drawings, but is not a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without inventive labor are still within the scope of protection of the present application.

Claims

1. An English spoken language recognition method, characterized by, The time dependence is calculated after dimension transformation of the time feature, and the time dependence feature is obtained by residual connection with the dimension-transformed time feature. The frequency dependence is calculated after dimension transformation of the frequency feature, and the frequency dependence feature is obtained by residual connection with the dimension-transformed frequency feature. The cross-attention mechanism is used to fuse and perform residual connection on the time dependence feature and the frequency dependence feature, and the fused feature is obtained. The voice signal is acquired and the audio feature is extracted, time feature and frequency feature are extracted from the audio feature; wherein the audio feature dimension is ; is the sample number, is the time dimension, is the frequency dimension; The fused feature is progressively down-sampled by a three-stage encoder; in the first stage, the fused feature is grouped along the time dimension by using the grouped multi-head self-attention mechanism, and attention is calculated for each group separately. Audio features that include both temporal and frequency characteristics The dimension of , is the channel dimension; the calculation process of the fusion feature is as follows: ； ； ； ； wherein, is a time-dependent feature; is a frequency-dependent feature; denotes a time feature dimension reshaping operation; denotes a frequency feature dimension reshaping operation; denotes an operation of a time domain module; denotes an operation of a frequency domain module; denotes an operation of a linear projection layer; denotes a cross-attention feature fusion operation; denotes a fused feature dimension reshaping operation; inputted the dimensions of the inputted and are reshaped to and , respectively, and then passed through the time domain module and the frequency domain module to calculate the time-dependent and frequency-dependent features, respectively, while connecting the input and output with a residual connection, and then passing the output through a linear layer to unify the dimensions to the same dimension, to obtain the time-dependent feature and the frequency-dependent feature , respectively; Time-dependent features are obtained through two linear projection layers. and frequency-dependent features Adjust the dimension downwards, and All dimensions are reshaped into Then through the cross-attention module By fusing time-dependent features and frequency-dependent features, the following features are obtained. , dimension ,feature It contains not only time-dependent information but also frequency-dependent information; it combines features Through linear projection layer The dimensions are reprojected back to their original size, and then the dimensions are reshaped to match the original size. Same dimensions Then, compare with the input. Performing residual connection operations yields the final output fused features. ; The output feature of the three-stage encoder is decoded to obtain the character probability distribution, and the speech recognition result is obtained therefrom. The feature extraction and down-sampling module is configured to acquire a speech signal and extract audio features, and extract time features and frequency features from the audio features.

2. The method of claim 1, wherein, The process of extracting time features and frequency features from the audio features includes: adopting a stacked The module constitutes a down-sampling module, and The channels of the module are sequentially increased, so as to extract the time features and the frequency features; specifically: The audio features are down-sampled in the time dimension and the frequency dimension by a convolution layer with a stride of , and then input into the first module after a batch normalization operation. First The output of the first module is input into the second module after passing through a convolution layer with a stride of and a batch normalization operation in sequence. ​ second The module's output is sequentially passed through steps with a step size of After the convolutional layer and batch normalization operation, the data is input into the third... In the module; Third The output of the module is dimensionally converted by a layer normalization operation and a linear layer operation in sequence to complete the extraction of time features and frequency features.

3. The method of claim 1, wherein, The grouped multi-head self-attention mechanism comprises: grouping along the time dimension by using the grouped multi-head self-attention mechanism, attention heads in each group sharing the same parameters, and the obtained attention matrix being respectively sent into the respective group attention heads to calculate attention after each group performing linear transformation, so that the attention complexity is reduced from to wherein, is the sequence length, is the feature dimension, is the number of groups, and the dimension of the attention matrix is changed from × to × , , .

4. The method of claim 1, wherein, The English oral language recognition method further comprises: the speech recognition result is obtained by processing by the trained speech recognition module, and the speech recognition module is trained by using a pre-training-fine-tuning paradigm, wherein parameters of a time-frequency dependency calculation process, parameters in a first stage encoder, and parameters of a decoding process are fine-tuned, and a uniform activation function is used. Activation function.

5. An English speech recognition system characterized by, The time-frequency dependence calculation module is configured to calculate the time dependence after dimension transformation of the time feature, and obtain the time dependence feature by residual connection with the dimension-transformed time feature. The frequency dependence is calculated after dimension transformation of the frequency feature, and the frequency dependence feature is obtained by residual connection with the dimension-transformed frequency feature. The cross-attention mechanism is used to fuse and perform residual connection on the time dependence feature and the frequency dependence feature, and the fused feature is obtained. The calculation process is as follows: The encoding module is configured to progressively down-sample the fused feature by using a three-stage encoder; in the first stage, the fused feature is grouped along the time dimension by using the grouped multi-head self-attention mechanism, and attention is calculated for each group separately. The decoding module is configured to decode the output feature of the three-stage encoder to obtain the character probability distribution, and obtain the speech recognition result therefrom. Audio features that include both temporal and frequency characteristics The dimension of , is the channel dimension; the calculation process of the fusion feature is as follows: ； ； ； ； wherein, is a time-dependent feature; is a frequency-dependent feature; denotes a time feature dimension reshaping operation; denotes a frequency feature dimension reshaping operation; denotes an operation of a time domain module; denotes an operation of a frequency domain module; denotes an operation of a linear projection layer; denotes a cross-attention feature fusion operation; denotes a fused feature dimension reshaping operation; inputted the dimensions of the inputted and are reshaped to and , respectively, and then passed through the time domain module and the frequency domain module to calculate the time-dependent and frequency-dependent features, respectively, while connecting the input and output with a residual connection, and then passing the output through a linear layer to unify the dimensions to the same dimension, to obtain the time-dependent feature and the frequency-dependent feature , respectively; Time-dependent features are obtained through two linear projection layers. and frequency-dependent features Adjust the dimension downwards, and All dimensions are reshaped into Then through the cross-attention module By fusing time-dependent features and frequency-dependent features, the following features are obtained. , dimension ,feature It contains not only time-dependent information but also frequency-dependent information; it combines features Through linear projection layer The dimensions are reprojected back to their original size, and then the dimensions are reshaped to match the original size. Same dimensions Then, compare with the input. Performing residual connection operations yields the final output fused features. ; The computer program is executed by the processor to complete the method of any one of claims 1-4. The computer program is executed by the processor to complete the method of any one of claims 1-4.

6. An electronic device, comprising: The computer program is executed by the processor to complete the method of any one of claims 1-4.

7. A computer readable storage medium characterized in that, ​ 8. A computer program product, characterised in that, ​

Images