Dialect speech recognition method and system based on Conformer neural network

Abstract

The invention provides a dialect speech recognition method and system based on a Conformer neural network, and the method comprises the steps: carrying out the preprocessing of input dialect speech data, and obtaining Fbank features; performing data enhancement on the Fbank features; performing normalization processing, feature dimension conversion and position coding on the enhanced Fbank features, and inputting the enhanced Fbank features into a Conformer encoder; the Conformer encoder extracts acoustic features through a plurality of Conformer encoder layers, carries out normalization processing on the acoustic features, and extracts feature information used for a decoder; the extracted feature information is processed and normalized through a bidirectional decoder structure, and the bidirectional decoder structure comprises a bidirectional decoder, a CTC decoding branch and an attention decoding branch; and carrying out weighted fusion on the output features of the CTC decoding branch and the attention decoding branch to obtain a final prediction result. And processing a prediction result through a greedy decoder to obtain an optimal text. The method not only considers the effective utilization of the local features of the dialect voice, but also considers the global interaction based on the content, and achieves a better dialect recognition effect.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of speech recognition technology, and in particular to a dialect speech recognition method and system based on a Conformer neural network. Background Art

[0002] With the rapid development of artificial intelligence (AI), speech recognition technology has been widely used in fields such as human-computer interaction and smart homes. However, current mainstream speech recognition systems are primarily optimized for Mandarin, resulting in relatively low accuracy for dialects. This is primarily due to the significant differences in intonation and phonetic features between dialects and Mandarin. Dialects often exhibit significant variations in speech rate, complex tones, and the frequent presence of polyphonetic characters. These characteristics make it difficult for traditional speech recognition models to adapt to the diverse speech characteristics of dialects, resulting in high misrecognition rates.

[0003] Traditional speech recognition methods are primarily based on hidden Markov models and Gaussian mixture model frameworks, achieving speech-to-text conversion by modeling the short-term features of speech signals. However, these methods are insufficiently sensitive to complex speech environments and dialect characteristics, and are unable to fully capture the long-term dependencies and global semantic information in speech signals. In recent years, the introduction of deep learning techniques, particularly convolutional neural networks and long short-term memory networks, has significantly improved speech recognition performance. However, while convolutional neural networks can effectively utilize local features, they are less effective at processing long-term dependencies. The subsequently proposed Transformer-based end-to-end speech recognition model, while performing well in processing long-term dependencies, is less than ideal when capturing local speech features due to its pure attention mechanism design. Summary of the Invention

[0004] In response to the above-mentioned problems existing in the prior art, the present invention provides a dialect speech recognition method and system based on Conformer, which can comprehensively extract acoustic features and improve the accuracy of recognition.

[0005] The technical solutions for achieving the purpose of the present invention are:

[0006] A dialect speech recognition method based on a Conformer neural network, comprising:

[0007] Step 1: preprocess the input dialect speech data to obtain Fbank features;

[0008] Step 2: Perform data enhancement on the Fbank features of the speech data;

[0009] Step 3: Normalize, transform and position-encode the enhanced Fbank features and input them into the Conformer encoder.

[0010] Step 4: The Conformer encoder extracts acoustic features through multiple Conformer encoder layers and performs normalization to extract feature information for the decoder;

[0011] Step 5: The feature information extracted in step 4 is processed and normalized by a bidirectional decoder structure, which includes a bidirectional decoder, a CTC decoding branch, and an attention decoding branch.

[0012] Step 6: Perform weighted fusion on the output features of the CTC decoding branch and the attention decoding branch to obtain the final prediction result;

[0013] Step 7: Process the prediction results through the greedy decoder to obtain the optimal text.

[0014] Furthermore, the speech data Fbank features are enhanced through noise enhancement, speech rate enhancement, volume enhancement, offset enhancement and SpecAugment.

[0015] Furthermore, step 3 specifically includes: normalizing the mean and standard deviation of the enhanced Fbank features, and transforming the dimensions of the normalized results through Conv2d subsampling, compressing the time domain features to 1 / 4, and finally encoding the position information and sending it to the Conformer encoder.

[0016] Furthermore, each encoder layer includes a first FeedForward module, a multi-head self-attention module, a convolution module, and a second FeedForward module, where:

[0017] The first Feed Forward module is a macaron-style feedforward neural network module. It adopts a sandwich structure, splitting the standard feedforward network into two halves and placing them at the beginning and end of the encoder layer. This module completes feature transformation through two linear transformation layers and a nonlinear activation function. Its output is scaled by 0.5 and then passed to the multi-head self-attention module.

[0018] The multi-head self-attention module projects the input features into the query and key-value spaces. Attention weights are calculated in parallel through multiple attention heads. Each attention head captures dependencies at different scales. The outputs of all attention heads are concatenated and linearly transformed to obtain the final attention features, which are then input into the convolution module.

[0019] The convolution module adopts a depth-wise separable convolution structure. It first expands the channel dimension through point-by-point convolution and uses gated linear units for feature selection. Then, it performs convolution operations in the time dimension and channel dimension through depth-wise separable convolution to complete the extraction of local features and output them to the second FeedForward module.

[0020] The second Feed Forward module has the same structure as the first Feed Forward module and is used to complete the final feature transformation.

[0021] Furthermore, the bidirectional decoder includes a token embedding layer, a positional encoding layer, and a multi-layer Conformer decoder. The feature processing specifically includes:

[0022] The input text sequence is first processed by the Token Embedding Layer, which converts the discrete text tokens into continuous vector representations using an embedding matrix of a predefined size.

[0023] The embedded vector is added to the positional encoding information. The positional encoding is generated by sine and cosine functions, which injects relative or absolute position information into each position in the sequence, allowing the model to perceive the positional relationship of the markers in the sequence.

[0024] The features then enter multiple layers of Conformer decoder for deep processing.

[0025] Furthermore, each Conformer decoder layer includes a self-attention module, an encoder-decoder attention module, and a feed-forward module. The deep processing process specifically includes:

[0026] The features are first processed by a self-attention module, which calculates the dependencies within the target sequence. Unlike the encoder, a masking mechanism is used to ensure that the current position can only see the previous position information.

[0027] Next is the encoder-decoder attention module processing, which establishes the association between the decoded features and the encoder output features, uses the decoded features as the query and the encoder output features as the key value, and obtains the acoustic information related to the current decoding position through attention calculation;

[0028] Finally, the Feed Forward module processes the features by transforming them through two linear transformation layers and activation functions to enhance the expressiveness of the features.

[0029] Furthermore, the CTC decoding branch generates a frame-level posterior probability distribution by modeling the acoustic features output by the bidirectional decryptor, and obtains a preliminary decoding sequence by removing repeated tokens and blank tokens.

[0030] Furthermore, the attention decoding branch processes the features output by the encoder based on the bidirectional decoder to generate another decoding sequence.

[0031] Furthermore, the output and input of each module in the Conformer encoder and the bidirectional decryptor are residually connected.

[0032] A dialect speech recognition system based on a Conformer neural network for implementing the method comprises:

[0033] The preprocessing unit is used to preprocess the input dialect speech data to obtain Fbank features;

[0034] Enhancement unit, used to perform data enhancement on the speech data Fbank features;

[0035] The encoder processing unit normalizes, transforms the feature dimensions, and position-encodes the enhanced Fbank features, and then inputs them into the Conformer encoder. The Conformer encoder extracts acoustic features through multiple Conformer encoder layers and normalizes them to extract feature information for the decoder.

[0036] The decoder processing unit is used to process and normalize the extracted feature information through a bidirectional decoder structure. The bidirectional decoder structure includes a bidirectional decoder, a CTC decoding branch, and an attention decoding branch.

[0037] The fusion unit is used to perform weighted fusion of the output features of the CTC decoding branch and the attention decoding branch to obtain the final prediction result;

[0038] The output unit processes the prediction results through a greedy decoder to obtain the optimal text.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The dialect speech recognition method based on the Conformer neural network provided by the present invention not only takes into account the effective utilization of the local features of the dialect speech, but also takes into account its global interaction based on content, thereby achieving a more outstanding dialect recognition effect; and by combining the strong representation ability of the Conformer architecture with a multi-branch decoding strategy, it significantly reduces insertion, deletion, and substitution errors in dialect speech recognition, and has high practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 Flowchart of the dialect speech recognition method based on Conformer neural network provided by an embodiment of the present invention.

[0043] Figure 2 A schematic diagram of the Conformer encoder flow provided in an embodiment of the present invention.

[0044] Figure 3 A schematic diagram of the Conformer decoder flow provided by an embodiment of the present invention.

[0045] Figure 4 A schematic diagram of the greedy decoder principle provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be arbitrarily combined with each other to form a feasible technical solution. This combination is not restricted by the sequence of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0047] The embodiment of the present invention provides a dialect speech recognition method based on Conformer neural network, see Figure 1 , the method comprising:

[0048] Step 1: Preprocess the input dialect speech data to obtain Fbank features;

[0049] Step 2: Perform data enhancement on the speech data by using noise enhancement, speech rate enhancement, volume enhancement, offset enhancement, and SpecAugment to enhance the dialect speech data.

[0050] Step 3: Normalize the Fbank features, transform the feature dimensions, position encode and input them into the Conformer encoder;

[0051] Step 4: The Conformer encoder includes multiple Conformer encoder layers. Each encoder layer includes a first feed forward module, a multi-head self-attention module, a convolution module, and a second feed forward module. A final normalization process is performed to extract feature information for the decoder.

[0052] Step 5: Through the bidirectional decoder, we obtain the CTC decoding branch and the attention decoding branch. The single decoder contains token embedding, position encoding, and multiple Conformer decoder layers. Each decoder layer includes a self-attention module, an encoder-decoder attention module, and a FeedForward module.

[0053] Step 6: Perform weighted fusion of the CTC branch and the attention branch to obtain the final prediction result;

[0054] Step 7: Send the result to the greedy decoder to obtain the optimal path and output the final text.

[0055] Based on the content of the above method embodiment, as an optional embodiment, the dialect speech recognition method based on the Conformer neural network provided in the embodiment of the present invention specifically includes step 2: performing data enhancement on the dialect speech data, mainly through the following two aspects: waveform-level data enhancement methods include: speed perturbation, volume perturbation, and noise injection; spectrum-level data enhancement methods include: frequency masking, time masking, etc.

[0056] Based on the content of the above method embodiment, as an optional embodiment, the dialect speech recognition method based on the Conformer neural network provided in the embodiment of the present invention, step 3 specifically includes: performing CMVN processing by loading the mean and standard deviation of the enhanced Fbank features, and transforming the dimension of the normalized results through Conv2d subsampling, compressing the features in the time domain to 1 / 4, and finally encoding the position information and sending it to the Conformer encoder.

[0057] Based on the content of the above method embodiment, as an optional embodiment, the dialect speech recognition method based on the Conformer neural network provided in the embodiment of the present invention specifically includes step 4: the features are then processed through multiple Conformer encoder layers. Each encoder layer contains four core modules, which are serially processed to extract rich acoustic features. The processing flow of each encoder layer is as follows: the first is a Macaron-style feedforward neural network module, which adopts a sandwich structure, splitting the standard feedforward network into two halves and placing them at the beginning and end of the encoder layer. This module completes feature transformation through two linear transformation layers and a nonlinear activation function. Its output is scaled by 0.5 and passed to the next module. The multi-head self-attention module projects the input features into the query and key spaces, and calculates attention weights in parallel using multiple attention heads. Each attention head captures dependencies at different scales. The outputs of all attention heads are concatenated and linearly transformed to obtain the final attention features, realizing the modeling of global context information. The convolution module adopts a depthwise separable convolution structure, first expanding the channel dimension through point-by-point convolution, and using gated linear units (GLUs) for feature selection. The subsequent depthwise separable convolution performs convolution operations in the time dimension and channel dimension respectively to complete the extraction of local features. This module also uses batch normalization and Swish activation function to process features. Finally, the features pass through the second feedforward neural network module with the same structure as the first module to complete the final feature transformation. During the processing of the entire encoder layer, the output of each submodule is residually connected to the input and processed using a normalization layer. After all encoder layers are processed, the features pass through the final normalization layer and output the acoustic feature representation for the decoder, resulting in:

[0058]

[0059]

[0060] Among them, FFN refers to the feedforward module, MHSA refers to the multi-head self-attention module, and Conv refers to the convolution module.

[0061] Based on the above method embodiments, as an optional embodiment, the dialect speech recognition method based on a Conformer neural network provided in the embodiments of the present invention specifically includes the following steps: The Conformer model adopts a bidirectional decoder structure, including decoders for both left-to-right and right-to-left directions, and cooperates with a CTC decoding branch to form a complete decoding system. At the beginning of decoding, the input text sequence is first processed by the Token Embedding Layer. This layer converts discrete text tokens into continuous vector representations using an embedding matrix of a predefined size. The embedded vectors are then added to the positional encoding information generated using sine and cosine functions, injecting relative or absolute position information for each position in the sequence, enabling the model to perceive the positional relationships between tokens in the sequence. The initialized features then enter the multi-layer decoder layer for deep processing. In each decoder layer, the features are first processed by the self-attention module. This module calculates dependencies within the target sequence. However, unlike the encoder, the self-attention in the decoder uses a masking mechanism to ensure that the current position only sees information from previous positions. This causal masking ensures the correctness of autoregressive decoding. Next, the encoder-decoder attention module establishes a correlation between the decoded features and the encoder output features. Specifically, the decoded features are used as queries, the encoder output features are used as keys, and attention calculations are performed to obtain acoustic information related to the current decoding position. This cross-modal attention mechanism enables the decoder to selectively extract information based on the importance of acoustic features. Finally, the feedforward neural network module transforms the features through two linear transformation layers and activation functions to enhance their expressiveness. The output of each submodule is residually connected to the input and passed through a normalization layer. The result is:

[0062]

[0063] SA refers to the self-attention module, EDA refers to the encoder-decoder attention module, and FFN refers to the feedforward neural network module. During bidirectional decoding, the left-to-right decoder decodes the text in normal order, while the right-to-left decoder processes the sequence in reverse order. The decoding results from both directions are ultimately weighted and fused.

[0064] Based on the content of the above method embodiment, as an optional embodiment, the dialect speech recognition method based on the Conformer neural network provided in the embodiment of the present invention, step 6 specifically includes: adopting a weighted fusion strategy of the CTC branch and the attention branch in the final prediction stage.

[0065] The CTC branch models the acoustic features output by the decoder to generate a frame-level posterior probability distribution and obtains a preliminary decoding sequence by removing repeated and blank tokens. The attention branch processes the features output by the encoder through a bidirectional decoder to generate another decoding sequence.

[0066] In the fusion phase, the model uses a preset weight coefficient to perform a weighted combination of the predicted probabilities of the two branches. For each possible output sequence, its final logarithmic probability is the weighted sum of the logarithmic probability of the CTC branch and the logarithmic probability of the attention branch.

[0067] The dialect speech recognition method based on the Conformer neural network provided by the embodiment of the present invention not only takes into account the effective utilization of the local features of the dialect speech, but also takes into account its global interaction based on content, thereby achieving a more outstanding dialect recognition effect; and by combining the strong representation ability of the Conformer architecture with the multi-branch decoding strategy, it significantly reduces the insertion, deletion and substitution errors in dialect speech recognition, and has high practical application value.

[0068] Specifically, the original Fbank features are first normalized using the global CMVN technique to eliminate variations between different audio data. Specifically, the features are normalized by calculating their mean and standard deviation to achieve a distribution with zero mean and unit variance. This step effectively improves the model's adaptability to diverse recording environments. The dimensionality conversion stage uses a two-dimensional convolutional subsampling layer, compressing the time-domain features to one-quarter their original size through two consecutive convolution operations. This not only reduces the computational complexity of subsequent processing but also extracts preliminary acoustic features by increasing the channel dimension. Subsequently, the positional encoding module adds positional information to each position in the feature sequence, enabling the model to perceive the relative or absolute positional relationships of elements within the sequence. The preprocessed features then enter multiple cascaded Conformer encoder layers, each of which contains four core functional modules. The first Feed Forward module, employing a Macaron architecture, performs a preliminary feature transformation via two linear transformation layers and an activation function. This is followed by a multi-head self-attention module, which projects the input features into multiple attention spaces and computes attention weights in parallel, effectively capturing long-range dependencies within the sequence. The third is the Convolution module, which uses a depth-separable convolution structure to extract local feature patterns through convolution operations in the time dimension and channel dimension. The design of this module not only maintains a low parameter amount, but also effectively extracts the local structural information of the acoustic features. Finally, there is the second Feed Forward module, which performs the final feature transformation. It should be noted that in each encoder layer, the output of each functional module will be residually connected to the input and normalized through LayerNormalization. This design not only facilitates lossless transmission of information, but also maintains the stability of deep network training. The series connection of multiple encoder layers enables the model to extract more abstract feature representations layer by layer. The specific flow chart is as follows Figure 2 .

[0069] The Conformer-based dialect speech recognition encoder described above encodes the text, generating a high-level feature representation that is then fed into the Conformer decoder. The Conformer model employs a bidirectional decoder (BiDecoder) architecture, consisting of two branches: forward decoding and backward decoding, and a CTC branch for decoding. At the beginning of decoding, the input text tokens are first processed by a token embedding layer. This layer maps discrete text tokens into a continuous vector space, generating a fixed-dimensional vector representation. This embedded vector is then added to the positional encoding, generated using sine and cosine functions. This positional encoding embeds positional information for each position in the sequence, enabling the model to distinguish tokens at different positions. The forward and backward decoding branches have the same architecture, but process the sequence in opposite directions. Each branch contains multiple decoder layers, each consisting of three key modules. The first is the Self-Attention Module, which calculates dependencies within the target sequence. Unlike the encoder's self-attention, the decoder uses a masking mechanism to ensure that the current position can only access information from previous positions, thus ensuring the causal nature of autoregressive decoding. Next is the Encoder-Decoder Attention Module, which establishes the association between the decoding features and the encoder output features. Specifically, the decoding features are used as queries and the encoder output features are used as keys to obtain the acoustic information related to the current decoding position through attention calculation. This cross-modal attention mechanism enables the decoder to extract selective information based on the importance of the acoustic features. The Feed Forward Module further transforms the features through two linear transformation layers and activation functions. The output of each module is residually connected to the input and passed through a normalization layer, which helps the training of deep networks and the transfer of information. The output of the bidirectional decoder works in parallel with the CTC branch. The CTC branch directly models the acoustic features of the encoder output and generates a posterior probability distribution at the frame level. The results of forward decoding and reverse decoding are fused in a weighted manner and jointly decoded with the output of the CTC branch. The specific process is as follows Figure 3 .

[0070] The greedy decoder is the basic decoding method in the Conformer model, which is mainly used to convert the probability distribution of the model output into the final text sequence. During the decoding process, for each time step, the greedy decoder selects the token with the highest probability as the output of the current position. Specifically, the decoder first receives the probability distribution matrix output by the model. Each row of the matrix corresponds to a time step, and each column corresponds to a token in the vocabulary. For each time step, the decoder selects the token index with the highest probability through the argmax operation and converts it into the corresponding text token. When processing the output of the CTC model, the greedy decoder also needs to perform deduplication, that is, remove consecutive repeated tokens and delete special blank tokens. In order to improve decoding efficiency, the greedy decoder usually adopts a batch processing method to process multiple sequences at the same time. After decoding is completed, the obtained token sequence is converted into the final text output, as shown in the following example. Figure 4 shown.

[0071] The following sections describe the various components of the Conformer algorithm in detail. The Feed Forward Module (FFM) is a Conformer algorithm used in both the encoder and decoder. In the Conformer encoder, each encoder layer contains two FFMs, each located at the beginning and end of the encoder layer using a macaron structure. In the decoder, each decoder layer contains one FFM, located at the end of the layer. The core of the FFM is a combination of two layers of linear transformations and a nonlinear activation function. Its mathematical expression can be described as follows:

[0072] FFN(x)=W2*σ(W1*x+b1)+b2 (8)

[0073] Where x is the input feature, W1 and W2 are weight matrices, b1 and b2 are bias terms, and σ is a nonlinear activation function. In Conformer, SiLU (Swish) is used as the activation function:

[0074] SiLU(x)=x*sigmoid(x) (9)

[0075] To enhance the model's feature extraction capabilities, the FFM expands the feature dimension by four times during the first linear transformation and then compresses it back to its original size during the second linear transformation. In the macaron-like structure, the outputs of both FFMs are scaled by a factor of 0.5. For the first FFM, the processing can be expressed as:

[0076] x=x+0.5*FFN(LayerNorm(x)) (10)

[0077] The same principle applies to the second FFM, where LayerNorm represents the normalization layer. Furthermore, the output of the FFM is residually connected to the input and processed by the normalization layer. The use of residual connections effectively prevents the vanishing gradient problem, while the normalization layer helps stabilize the training process of deep networks. The complete processing flow can be expressed as:

[0078] y=LayerNorm(x+Dropout(FFN(x))) (11)

[0079] The Convolution Module enhances the model's ability to model local features by introducing convolution operations. This module uses the design concept of depthwise separable convolution to achieve efficient feature extraction. The process begins with a normalization layer to standardize the input features. Subsequently, a point-by-point convolution is used to expand the number of feature channels to twice the original number. The purpose of this expansion is to provide sufficient feature dimensions for subsequent gated linear unit (GLU) operations. The core idea of GLU is to use half of the features as a gating signal to control the flow of the other half of the features. Its mathematical expression can be expressed as:

[0080]

[0081] Among them, x1 and x2 are the front and back halves of the input features respectively. Denotes element-wise multiplication, and σ is the sigmoid activation function. The features processed by GLU enter the depthwise separable convolution layer, which is the core component of the convolution module. Depthwise separable convolution first performs depthwise convolution on each channel independently, and then fuses information between channels through point-by-point convolution. Its basic form can be expressed as:

[0082] DSConv(x)=PointwiseConv(DepthwiseConv(x)) (13)

[0083] During the depthwise convolution stage, a causal convolution design is used to ensure that the output at the current moment depends only on the input at the current and previous moments. To enhance nonlinear expressiveness, batch normalization and the Swish activation function are applied after the depthwise convolution. The convolution module ends with another pointwise convolution, which compresses the feature dimensions back to their original size. The entire module also includes a residual connection that adds the input features to the processed features, facilitating gradient backpropagation and preserving deep features.

[0084] The Multi-Headed Self-Attention Module is a key component of the Conformer model responsible for capturing long-range dependencies. This module uses multiple attention heads in parallel to extract correlations between sequence features from different representation spaces. Before processing begins, input features are first normalized and then linearly transformed into three different representation spaces: query, key, and value. For each attention head, these three projections have the same dimension of d_model / h.

[0085] Where: d_model is the feature dimension of the model, h is the number of attention heads. The core calculation of multi-head self-attention can be expressed as:

[0086] MultiHead(Q,K,V)=Concat(head1,...,head h )W O (14)

[0087] The calculation of each attention head follows the scaled dot product attention mechanism:

[0088]

[0089] In this formula, W i Q 、W i K 、W i V is the projection matrix of each head, d k is the dimension of the key vector. The purpose of the division is to prevent the dot product from being too large, which would cause the softmax gradient to vanish. In practice, multi-head attention allows the model to simultaneously focus on different positions in the sequence, with each attention head potentially focusing on different types of dependencies. For example, some heads may focus on local relationships between adjacent positions, while others may capture long-range semantic connections. This parallel multi-view processing mechanism significantly enhances the model's expressive power. To improve training efficiency and prevent overfitting, a masking mechanism for the attention scores is first implemented. A causal mask used in the decoder ensures that the current position can only access information from previous positions. Second, dropout is applied. Dropout layers are added to both the attention weights and the final output to effectively prevent overfitting. The output of the multi-head attention is linearly transformed and then residually connected to the original input. Finally, the entire processing result is passed through a normalization layer to ensure the distribution stability of the output features.

[0090] This invention not only captures global contextual information through a self-attention mechanism but also effectively extracts local features through convolution operations. Its encoder utilizes a multi-layered Conformer block structure, with each block comprising a multi-head self-attention module, a convolution module, and a feedforward neural network module, enabling comprehensive acoustic feature extraction. During the decoding phase, the model utilizes a bidirectional decoder structure, combined with CTC loss and an attention mechanism, to improve recognition accuracy.

[0091] The implementation basis of each embodiment of the present invention is achieved through programmatic processing of the system. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention can be encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides a dialect speech recognition system based on a Conformer neural network. This system is used to implement the dialect speech recognition method based on a Conformer neural network in the above method embodiment. The system includes:

[0092] The preprocessing unit is used to preprocess the input dialect speech data to obtain Fbank features;

[0093] Enhancement unit, used to perform data enhancement on the speech data Fbank features;

[0094] The encoder processing unit normalizes, transforms the feature dimensions, and position-encodes the enhanced Fbank features, and then inputs them into the Conformer encoder. The Conformer encoder extracts acoustic features through multiple Conformer encoder layers and normalizes them to extract feature information for the decoder.

[0095] The decoder processing unit is used to process and normalize the extracted feature information through a bidirectional decoder structure. The bidirectional decoder structure includes a bidirectional decoder, a CTC decoding branch, and an attention decoding branch.

[0096] The fusion unit is used to perform weighted fusion of the output features of the CTC decoding branch and the attention decoding branch to obtain the final prediction result;

[0097] The output unit processes the prediction results through a greedy decoder to obtain the optimal text.

[0098] It should be noted that, in addition to being used to implement the methods in the above-mentioned method embodiments, the system in the system embodiments provided by the present invention can also be used to implement the methods in other method embodiments provided by the present invention. The only difference lies in the setting of corresponding functional modules, and its principle is basically the same as the principle of the above-mentioned device embodiments provided by the present invention. As long as those skilled in the art refer to the specific technical solutions in other method embodiments on the basis of the above-mentioned device embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and on the premise of ensuring the practicality of the technical solutions, they can improve the system in the above-mentioned system embodiments to obtain corresponding system embodiments for implementing the methods in other method embodiments.

[0099] It should be noted that the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, the elements defined by the phrase "comprise..." do not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising the elements.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A dialect speech recognition method based on Conformer neural network, characterized in that: include: Step 1: preprocess the input dialect speech data to obtain Fbank features; Step 2: Perform data enhancement on the Fbank features of the speech data; Step 3: Normalize, transform and position-encode the enhanced Fbank features and input them into the Conformer encoder. Step 4: The Conformer encoder extracts acoustic features through multiple Conformer encoder layers and performs normalization to extract feature information for the decoder; Step 5: The feature information extracted in step 4 is processed and normalized by a bidirectional decoder structure, which includes a bidirectional decoder, a CTC decoding branch, and an attention decoding branch. Step 6: Weightedly fuse the output features of the CTC decoding branch and the attention decoding branch to obtain the final prediction result; Step 7: Process the prediction results through a greedy decoder to obtain the optimal text.

2. The dialect speech recognition method based on Conformer neural network according to claim 1, characterized in that: The Fbank features of speech data are enhanced through noise enhancement, speech rate enhancement, volume enhancement, offset enhancement and SpecAugment.

3. The dialect speech recognition method based on Conformer neural network according to claim 1, characterized in that: The step 3 specifically includes: normalizing the mean and standard deviation of the enhanced Fbank features, and transforming the dimensions of the normalized results through Conv2d subsampling, compressing the features in the time domain to 1 / 4, and finally encoding the position information and sending it to the Conformer encoder.

4. The dialect speech recognition method based on Conformer neural network according to claim 1, characterized in that: Each encoder layer consists of a first FeedForward module, a multi-head self-attention module, a convolution module, and a second FeedForward module, where: The first Feed Forward module is a macaron-style feedforward neural network module. It adopts a sandwich structure, splitting the standard feedforward network into two halves and placing them at the beginning and end of the encoder layer. This module completes feature transformation through two linear transformation layers and a nonlinear activation function. Its output is scaled by 0.5 and then passed to the multi-head self-attention module. The multi-head self-attention module projects the input features into the query and key-value spaces. Attention weights are calculated in parallel through multiple attention heads. Each attention head captures dependencies at different scales. The outputs of all attention heads are concatenated and linearly transformed to obtain the final attention features, which are then input into the convolution module. The convolution module adopts a depthwise separable convolution structure. It first expands the channel dimension through point-by-point convolution and uses gated linear units for feature selection. Then, it performs convolution operations in the time dimension and channel dimension through depthwise separable convolution to complete the extraction of local features and output them to the second Feed Forward module. The second Feed Forward module has the same structure as the first Feed Forward module and is used to complete the final feature transformation.

5. The dialect speech recognition method based on Conformer neural network according to claim 1, characterized in that: The bidirectional decoder includes a token embedding layer, a positional encoding layer, and a multi-layer Conformer decoder. The feature processing specifically includes: The input text sequence is first processed by the Token Embedding Layer, which converts the discrete text tokens into continuous vector representations using an embedding matrix of a predefined size. The embedded vector is added to the positional encoding information. The positional encoding is generated by sine and cosine functions, which injects relative or absolute position information into each position in the sequence, allowing the model to perceive the positional relationship of the markers in the sequence. The features then enter multiple layers of Conformer decoder for deep processing.

6. The dialect speech recognition method based on Conformer neural network according to claim 5, characterized in that: Each Conformer decoder layer includes a self-attention module, an encoder-decoder attention module, and a FeedForward module. The deep processing process specifically includes: The features are first processed by a self-attention module, which calculates the dependencies within the target sequence. Unlike the encoder, a masking mechanism is used to ensure that the current position can only see the previous position information. Next is the encoder-decoder attention module processing, which establishes the association between the decoded features and the encoder output features, uses the decoded features as the query and the encoder output features as the key value, and obtains the acoustic information related to the current decoding position through attention calculation; Finally, the Feed Forward module processes the features by transforming them through two linear transformation layers and activation functions to enhance the expressiveness of the features.

7. The dialect speech recognition method based on Conformer neural network according to claim 1, characterized in that: The CTC decoding branch generates a frame-level posterior probability distribution by modeling the acoustic features output by the bidirectional decryptor, and obtains a preliminary decoding sequence by removing repeated tokens and blank tokens.

8. The dialect speech recognition method based on Conformer neural network according to claim 1, characterized in that: The attention decoding branch processes the features output by the encoder based on the bidirectional decoder to generate another decoded sequence.

9. The dialect speech recognition method based on Conformer neural network according to claim 1, characterized in that: The output and input of each module in the Conformer encoder and the bidirectional decryptor are residually connected.

10. A dialect speech recognition system based on a Conformer neural network that implements the method according to any one of claims 1 to 9, characterized in that: include: The preprocessing unit is used to preprocess the input dialect speech data to obtain Fbank features; Enhancement unit, used to perform data enhancement on the speech data Fbank features; The encoder processing unit normalizes, transforms the feature dimensions, and position-encodes the enhanced Fbank features, and then inputs them into the Conformer encoder. The Conformer encoder extracts acoustic features through multiple Conformer encoder layers and normalizes them to extract feature information for the decoder. The decoder processing unit is used to process and normalize the extracted feature information through a bidirectional decoder structure. The bidirectional decoder structure includes a bidirectional decoder, a CTC decoding branch, and an attention decoding branch. The fusion unit is used to perform weighted fusion of the output features of the CTC decoding branch and the attention decoding branch to obtain the final prediction result; The output unit processes the prediction results through a greedy decoder to obtain the optimal text.

Images