Multi-dialect speech recognition method based on Transform self-attention mechanism

Abstract

The invention discloses a multi-dialect speech recognition method based on a Transform self-attention mechanism, and belongs to the technical field of speech recognition, and the method comprises the steps: obtaining a multi-dialect audio data set, and carrying out the preprocessing and feature extraction; using a dialect topological space mapping module to construct topological space representation of dialect features; inputting the topological mapping features into a Transform encoder, and performing encoding processing by using a cross-dialect attention fusion mechanism and a dialect feature dynamic weighting mechanism; according to the method, Riemannian manifold mapping, graph attention fusion and gating dynamic weighting are introduced, the space structure relation of the speech features among dialects is accurately modeled, the recognition accuracy under a multi-dialect scene is remarkably improved, the word error rate is averagely reduced by 18%-25%, and the recognition accuracy of a few dialects is improved by 30%-40%.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, specifically to a multi-dialect speech recognition method based on the Transformer self-attention mechanism. It is particularly suitable for handling multi-dialect input scenarios with significant differences in speech features, and achieves high-precision speech recognition through dialect topology space mapping and cross-dialect attention fusion mechanism. Background Technology

[0002] With the development of artificial intelligence technology, speech recognition has become an important means of human-computer interaction. In existing technology, Chinese patent CN 114783418 A discloses an end-to-end speech recognition method and system based on a sparse self-attention mechanism. This patent samples the Query matrix, recalculates the attention weights using the sampled matrix and the downsampled Key matrix, and associates the recalculated attention weights with the Value matrix after replacing it with a mean vector to obtain a sparse self-attention relationship. Specifically, this method performs convolutional downsampling on the input acoustic features to obtain the Key matrix, calculates the vector with the largest value in each dimension of the input acoustic feature sequence as the global vector, calculates the attention weight matrix using the global vector and the original Query matrix, then samples the Query matrix, extracts the query vectors corresponding to the λ indices with the largest weights in the attention weight matrix, and recalculates the attention weights using the sampled Query matrix and Key matrix. This method reduces the number of dot product operations without increasing model complexity, improving storage and time efficiency.

[0003] However, the aforementioned existing technologies have significant shortcomings when handling multi-dialect speech recognition tasks. Specifically, in multi-dialect speech recognition scenarios, there are significant differences in acoustic features between different dialects, including differences in phoneme distribution, prosodic patterns, and pronunciation methods. The sparse attention mechanism used in existing technologies mainly reduces computational load through query sampling and global vector computation, but this mechanism lacks the ability to model the spatial structural relationships of speech features between dialects. When the input speech comes from dialects with significant differences in speech features (such as Cantonese, Wu, and Minnan), due to the fundamental differences in the distribution and topological relationships of acoustic features in the feature space of different dialects, a simple global maximum vector cannot effectively capture the dialect-specific speech patterns. This leads to the attention weight calculation being biased towards dialects with a higher proportion in the training data, resulting in a significant decrease in speech recognition accuracy for a few dialects or dialect boundary regions, with the error rate potentially increasing by 20% to 35%. Furthermore, the mean vector replacement strategy in the existing technology uses the global mean for the query vector that has been dropped out. This approach is acceptable in a single dialect scenario, but when multiple dialects are mixed, the global mean will lead to ambiguity and confusion of dialect features, making it impossible to maintain the uniqueness and distinguishability of each dialect feature, and further reducing the system's ability to adapt to dialect differences. Summary of the Invention

[0004] To address the technical problems in existing technologies for multi-dialect speech recognition, such as insufficient modeling of the spatial structural relationships of speech features between dialects, insensitivity of attention weight calculation to dialect differences, and ambiguity of dialect features caused by feature processing, the present invention aims to provide a multi-dialect speech recognition method based on the Transformer self-attention mechanism. By introducing a dialect topological space mapping mechanism, a cross-dialect attention fusion mechanism, and a dialect feature dynamic weighting mechanism, the spatial structural relationships between dialects are fully considered in the feature extraction stage, enabling the model to accurately capture and effectively fuse the speech features of different dialects, thereby significantly improving the speech recognition accuracy in multi-dialect scenarios.

[0005] To achieve the above-mentioned objectives, this invention provides a multi-dialect speech recognition method based on the Transformer self-attention mechanism, comprising the following steps:

[0006] Step S1: Obtain a multi-dialect audio dataset, preprocess and extract features from the audio dataset to obtain the original acoustic feature sequence;

[0007] Step S2: The original acoustic feature sequence is processed using the dialect topology space mapping module to construct a topology space representation of the dialect features and obtain a topology mapping feature sequence.

[0008] Step S3: Input the topology mapping feature sequence into the Transformer encoder. In the Transformer encoder, the attention weights are calculated using a cross-dialect attention fusion mechanism, and the features are weighted using a dialect feature dynamic weighting mechanism to obtain the encoded feature sequence.

[0009] Step S4: Input the encoded feature sequence into the decoder for decoding to obtain the corresponding text sequence.

[0010] Preferably, the dialect topology space mapping module adopts a dialect feature mapping method based on Riemannian manifolds to map the acoustic features of different dialects into a unified topology space, and calculates the similarity between dialects through geodesic distance.

[0011] Preferably, the cross-dialect attention fusion mechanism constructs a dialect relationship graph and considers the phonetic distance and topological adjacency between dialects when calculating attention weights, thereby realizing cross-dialect feature interaction and information transmission.

[0012] Preferably, the dialect feature dynamic weighting mechanism dynamically adjusts the feature weights based on the dialect confidence of the input speech, assigning higher weights to regions with significant dialect features, thereby enhancing the model's sensitivity to dialect differences.

[0013] Preferably, the construction process of the dialect topology space mapping module includes: first, training a dialect classifier using dialect annotation data to extract prototype feature vectors of different dialects; then, constructing a dialect topology graph in the feature space, where nodes represent dialect prototypes and edge weights represent the phonetic distance between dialects; and finally, defining a manifold structure using a Riemannian metric tensor to map acoustic features to the manifold space.

[0014] Preferably, the calculation process of the cross-dialect attention fusion mechanism includes: performing a linear transformation on the input topological mapping features to obtain a Query matrix, a Key matrix, and a Value matrix; constructing an adjacency matrix based on the dialect relationship graph, which encodes the topological adjacency information between dialects; combining standard dot product attention with graph attention based on the adjacency matrix when calculating the attention score to obtain the fused attention weight; and finally using the attention weight to perform a weighted summation on the Value matrix to obtain the cross-dialect fused feature representation.

[0015] Preferably, the dialect feature dynamic weighting mechanism adopts a gating mechanism, dynamically generating weight coefficients based on the dialect confidence vector, applying different weights to features at different time steps, and the calculation of the weight coefficients integrates the dialect prediction probability of the current frame and the dialect distribution statistics of historical frames.

[0016] Preferably, the preprocessing includes audio resampling, pre-emphasis, framing, and windowing. The feature extraction uses Mel frequency cepstral coefficients or log-Mel filter bank features, and the feature dimension is set to 80 to 256 dimensions.

[0017] Preferably, the Transformer encoder comprises 6 to 12 encoder layers, each encoder layer including a multi-head self-attention sub-layer, a feedforward neural network sub-layer, residual connections, and layer normalization, with the number of multi-head attention sub-heads set to 4 to 16.

[0018] Preferably, the decoder is a sequence-to-sequence decoder based on an attention mechanism, the decoding process uses a beam search algorithm, the beam width is set to 5 to 20, and the number of decoder layers is the same as or half the number of encoder layers.

[0019] This invention, by introducing a dialect topological space mapping mechanism, a cross-dialect attention fusion mechanism, and a dynamic weighting mechanism for dialect features, has the following significant advantages over existing technologies:

[0020] First, the dialect topological space mapping mechanism, based on Riemannian manifold theory, maps the acoustic features of different dialects to a unified topological space. By measuring the similarity between dialects through geodesic distance, it effectively preserves the spatial structural information of dialect features. Compared to the simple global maximum vector of existing technologies, this invention can more accurately capture the complex relationships between dialects, making the model more sensitive to dialect differences.

[0021] Second, the cross-dialect attention fusion mechanism constructs a dialect relationship graph and combines it with a graph attention mechanism. When calculating attention weights, it considers both the similarity of speech features and the phonetic distance between dialects, thus achieving effective information transfer across dialects. This mechanism allows the model to draw on feature information from similar dialects when processing a particular dialect, significantly improving its ability to recognize a few dialects and dialect boundary regions.

[0022] Third, the dialect feature dynamic weighting mechanism employs a gating mechanism, dynamically adjusting feature weights based on dialect confidence levels, assigning higher weights to regions with significant dialect features. This adaptive weighting strategy avoids the dialect feature ambiguity problem caused by global mean in existing technologies, enhancing the model's ability to learn specific dialect speech patterns.

[0023] Fourth, the overall technical solution of this invention significantly improves the recognition accuracy in multi-dialect scenarios while maintaining computational efficiency. On a test set containing more than 10 Chinese dialects, compared with the prior art, the word error rate of this invention is reduced by an average of 18% to 25%, with a particularly significant improvement in recognition accuracy for a few dialects, reaching 30% to 40%.

[0024] Fifth, the method of the present invention has good scalability and generalization ability. When a new dialect is added, only the dialect topology graph needs to be updated and a small number of dialect-specific parameters need to be trained. There is no need to retrain the entire model, which greatly reduces the system maintenance cost. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the overall process of the multi-dialect speech recognition method based on the Transformer self-attention mechanism of the present invention;

[0026] Figure 2 This is a schematic diagram of the dialect topology space mapping module of the present invention;

[0027] Figure 3 This is a schematic diagram of the computational process of the cross-dialect attention fusion mechanism of the present invention;

[0028] Figure 4 This is a schematic diagram illustrating the working principle of the dialect feature dynamic weighting mechanism of this invention. Detailed Implementation

[0029] Please refer to Figures 1-4 The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments to make the technical solution of the present invention clearer. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the scope of protection of the present invention.

[0030] like Figure 1 As shown, this invention provides a multi-dialect speech recognition method based on the Transformer self-attention mechanism. This method addresses technical challenges in multi-dialect speech recognition scenarios, such as significant differences in speech features between dialects and the insensitivity of attention mechanisms to dialect differences. By introducing innovative technologies such as dialect topology space mapping, cross-dialect attention fusion, and dynamic weighting of dialect features, it achieves high-precision recognition of multi-dialect speech. The entire method's technical flow includes four main stages: audio data preprocessing and feature extraction, dialect topology space mapping, cross-dialect attention-based encoding processing, and text decoding output. In practical applications, this method can handle speech input from different dialect regions, effectively addressing complex scenarios such as dialect mixing and dialect boundaries, significantly improving the practicality and robustness of speech recognition systems in multi-dialect environments in China.

[0031] Step S1, Audio data preprocessing and feature extraction:

[0032] In the first step of this invention, a multi-dialect audio dataset is first acquired. Preferably, this dataset should cover speech samples from major dialect regions in China, including but not limited to Mandarin, Cantonese, Wu, Minnan, Hakka, Xiang, and Gan dialects. The audio samples for each dialect should include speech from different speakers, genders, and age groups to ensure data diversity and representativeness. The audio files in the dataset are typically in WAV or MP3 format, with a sampling rate of 16kHz to 48kHz and a quantization bit depth of 16 bits.

[0033] The acquired raw audio data requires a series of preprocessing operations to improve the effectiveness of subsequent processing. Specifically, the preprocessing includes the following steps: First, all audio files are resampled, with the sampling rate standardized to 16kHz. This is a commonly used standard sampling rate in speech recognition tasks, preserving sufficient speech information while controlling the data volume. Then, the resampled audio signal undergoes pre-emphasis processing. The purpose of pre-emphasis is to compensate for high-frequency components, making the speech signal's spectrum flatter. The digital filter used for pre-emphasis is a first-order finite impulse response filter, whose transfer function can be expressed as... ,in The pre-emphasis coefficient ranges from 0.95 to 0.98, with a preferred value of 0.97. Next, the pre-emphasized signal is framed, with a frame length of 25 milliseconds and a frame shift of 10 milliseconds. There is a 15-millisecond overlap between adjacent frames to ensure the continuity of the speech signal. A Hamming window function is used for framing; the mathematical expression of the window function is:

[0034] ,

[0035] in: For the Hamming window function at the th The value of each sampling point; The window length is equal to the number of sampling points corresponding to the frame length. At a sampling rate of 16kHz, ; This is the sampling point index, with values ​​ranging from 0 to... The constants 0.54 and 0.46 are the standard coefficients of the Hamming window; Pi is approximately 3.14159.

[0036] After preprocessing, feature extraction is performed on each frame of the speech signal. This invention preferably uses log-Mel filter bank features as acoustic features, which can better simulate the human ear's perception of speech. The specific steps of feature extraction include: first, performing a Fast Fourier Transform on the windowed signal of each frame to obtain the spectrum; then, calculating the power spectral density; next, passing the power spectrum through a set of Mel-scale triangular filter banks, with the number of filter banks set to 80 to 256, preferably 128; finally, taking the logarithm of the filter bank output to obtain the log-Mel filter bank features. The features of each frame can be represented as a... A dimensional vector, where This is equal to the number of filter banks. Assume a segment of speech is divided into frames to obtain... If the frame is defined, then the original acoustic feature sequence of the entire speech can be represented as a matrix. ,in For frame number, This is the feature dimension. In a preferred embodiment of the invention, For a 3-second audio clip, with a frame shift of 10 milliseconds, the number of frames is... Approximately 300.

[0037] Step S2, Dialect topological space mapping:

[0038] like Figure 2 As shown, the dialect topology space mapping module is one of the core innovations of this invention. This module maps acoustic features from different dialects into a unified topological space, enabling accurate representation of the phonological feature relationships between dialects. Traditional methods typically process multi-dialect data directly in the original feature space, ignoring the distribution differences and topological structures of different dialects within the feature space, resulting in insufficient modeling ability for dialect differences. This invention introduces Riemannian manifold theory, treating the dialect feature space as a Riemannian manifold, and defining appropriate metrics on this manifold, allowing the distance between dialects to more accurately reflect their phonological similarity.

[0039] The construction of the dialect topological space mapping module consists of three sub-steps: dialect prototype extraction, dialect topological graph construction, and Riemannian manifold mapping. In the first sub-step, a dialect classifier is trained using labeled dialect data. Specifically, a dataset containing... This dataset contains labeled datasets for several dialects, each containing several speech samples and their dialect labels. Convolutional neural networks or recurrent neural networks are used as the basic architecture for the dialect classifier, with the input being the original acoustic feature sequence. The output is A dialect probability distribution vector is generated. During training, the cross-entropy loss function is used, and the classifier parameters are optimized through backpropagation. After training, for each dialect, the feature representation of the penultimate layer of the classifier is extracted from all training samples of that dialect, and the mean of these feature representations is calculated to obtain the prototype feature vector of that dialect. ,in Indicates a dialect index. The dimension of the prototype feature is preferably 256 to 512 dimensions. The prototype feature vectors constitute the dialect prototype set. Each prototype represents the core phonetic features of the corresponding dialect.

[0040] In the second sub-step, a dialect topology graph is constructed based on the dialect prototype set. The dialect topology graph is a weighted undirected graph, denoted as [dialect topology graph]. ,in For a set of nodes, each node Corresponding to a dialect; Let it be the set of edges; For an adjacency matrix, matrix elements Indication of dialect and dialects The adjacency matrix is ​​constructed using similarity calculations based on prototype features, determining the phonological distance or similarity between the two dialects. and dialects Calculate its prototype feature vector and Cosine similarity between them:

[0041] ,

[0042] in: dialect and dialects The cosine similarity between them ranges from 1 to 10. arrive ; and Dialects and dialects The prototype feature vectors, all of which have dimensions. ; express Transpose of; and Representing vectors respectively Norm, also known as Euclidean norm, is calculated using the following formula: ,in For vectors The Each component.

[0043] Based on cosine similarity, the elements of the adjacency matrix are defined as follows:

[0044] ,

[0045] in: Adjacency matrix The Middle Line number The elements of the column represent dialects. and dialects Connection weights between them; The cosine similarity calculated earlier; The bandwidth parameter of the Gaussian kernel controls the mapping relationship between similarity and weight, and its value ranges from 0.1 to 0.5, with an optimal value of 0.3. The similarity threshold is used to determine which dialect pairs have edge connections; its value ranges from 0.3 to 0.7, with a preferred value of 0.5. (Exponential function) Mapping similarity differences to a weight space; in the molecule This represents the cosine distance; the higher the similarity, the smaller the distance, and the greater the weight.

[0046] The way this adjacency matrix is ​​constructed ensures that dialects with similar phonology are connected in the graph by edges with larger weights, while dialects with large differences have no edge connections or edge weights close to zero, thus accurately characterizing the topological adjacency relationships between dialects.

[0047] In the third sub-step, the manifold structure is defined using the Riemannian metric tensor, mapping the original acoustic features to the Riemannian manifold space. A Riemannian manifold is a differential manifold with an inner product structure, on which geodesic distances can be defined to measure the shortest path length between two points. For the input sequence of original acoustic features... First, it is mapped to the tangent space of the manifold through a learnable linear transformation:

[0048] ,

[0049] in: The mapped feature sequence is also called the tangent space representation; The input is the original acoustic feature sequence. For frame number, The original feature dimension; The weight matrix is ​​the linear transformation matrix. The feature dimensions are mapped, and this matrix is ​​obtained through training. This is the bias vector, which is applied to each frame via a broadcast mechanism in actual computation; addition operation. Perform matrix multiplication first, then add the bias.

[0050] Obtain the tangent space representation Next, a Riemannian metric tensor needs to be defined on the manifold. The Riemannian metric tensor is a positive definite symmetric matrix used to define the inner product and distance on the manifold. In this invention, the Riemannian metric tensor is dynamically generated through the relationship between dialect prototypes and input features. For the... Frame characteristics (Right now The (line), calculate its relationship with each dialect prototype. The similarity is used to obtain the dialect affiliation probability distribution:

[0051] ,

[0052] in: For the first Frame features of dialect The probability of belonging to a dialect ranges from 0 to 1, and the sum of the probability of belonging to all dialects equals 1. For the first The slicing space representation of a frame has a dimension of ; dialect The prototype feature vector, with dimension . ; This is the vector inner product, and the result is a scalar representing the similarity between the feature and the prototype. The temperature parameter is used to control the smoothness of the softmax function, with a value ranging from 0.1 to 1.0, and a preferred value of 0.5. The total number of dialects; the summation symbol in the denominator. Indicates all The dialects are accumulated for normalization; It is an exponential function that maps the inner product similarity to a positive number space.

[0053] Based on dialect affiliation probability, construct the first... Riemannian metric tensor of a frame :

[0054] ,

[0055] in: For the first The Riemannian metric tensor corresponding to a frame is a A positive definite symmetric matrix of dimension 1; Total number of dialects; For the previously calculated first... Frame Dialect The probability of belonging; dialect The corresponding base metric matrix is ​​a learnable parameter that is optimized through backpropagation during training. The summation symbol represents a weighted summation over all dialects, with the weights being the attribution probabilities. This formula enables the dynamic generation of the metric tensor, adaptively adjusting the metric method based on the dialect attribution of the input features.

[0056] Using the Riemannian metric tensor, the geodesic distance between two eigenvectors on a manifold can be calculated. For the ... Frame and the Frame characteristics and Its distance on the manifold can be approximated as:

[0057] ,

[0058] in: For the first Frame and the The geodesic distance of the frame on the Riemannian manifold, in units related to the units of the feature; and The first Frame and the The slicing space representation of a frame has dimensions of 1. ; The vector difference represents the relative positions of the features in the tangent space between the two frames; For the first The Riemannian metric tensor of the frame, with dimension . ; This is the transpose of the vector difference; It is a quadratic form, and the calculation result is a scalar; For square root operations, ensure that the distance is a non-negative real number.

[0059] Through the Riemannian manifold mapping described above, the original acoustic feature sequence Mapped to topological mapping feature sequence This sequence not only contains the acoustic information of the speech but also embeds the topological structure information of the dialect. This mapping method enables subsequent attention calculations to be based on more accurate dialect feature representations, significantly improving the model's ability to model multi-dialect speech.

[0060] In practical implementation, the dialect topology space mapping module can be used as a pre-module of the Transformer encoder, and its parameters (including linear transformation weights) are... Bias Dialect Prototype and the fundamental metric matrix The model is jointly trained end-to-end with the entire system, optimized using backpropagation and gradient descent. A multi-task learning strategy is employed during training to simultaneously optimize the word error rate for speech recognition and the accuracy of dialect classification, ensuring that the mapping module retains the speech content information required for speech recognition while capturing dialect-specific speech patterns.

[0061] Step S3, encoding processing based on cross-dialect attention fusion mechanism:

[0062] The topological mapping feature sequence obtained in step S2 The input is fed into a Transformer encoder for deep feature encoding. The Transformer encoder of this invention consists of multiple stacked encoder layers. Each encoder layer includes a cross-dialect attention fusion sublayer, a feedforward neural network sublayer, residual connections, and layer normalization. The cross-dialect attention fusion mechanism is the second core innovation of this invention. Based on the standard self-attention mechanism, this mechanism incorporates topological information from the dialect relation graph, enabling attention calculation to consider not only the phonological similarity between features but also the phonological distance and topological adjacency between dialects, thereby achieving cross-dialect feature interaction and information transfer.

[0063] like Figure 3 As shown, the computation process of the cross-dialect attention fusion mechanism is as follows: First, the input topological mapping feature sequence is processed. Perform a linear transformation to obtain the query matrix. Key matrix Sum matrix :

[0064] ,

[0065] in: For querying the matrix, For sequence length, For query vector dimensions; The key matrix has the same dimensions as the query matrix; For value matrices, The dimension of the value vector; Given the input topological mapping feature sequence, For feature dimensions; The linear transformation weights of the query matrix are learnable parameters. The linear transformation weights of the key matrix are learnable parameters. The weights are linear transformations of the value matrix and are learnable parameters; in multi-head attention mechanisms, they are typically set to... ,in The number of attention heads.

[0066] Then, the standard dot product attention score matrix is ​​calculated. :

[0067] ,

[0068] in: The dot product attention score matrix has dimensions of . Matrix elements Indicates the first The query vector pairs the first Attention score of each key vector; For querying the matrix, the dimension is ; The transpose of the key matrix has dimensions of . ; This is a matrix multiplication, and the result dimension is... ; This is a scaling factor used to prevent the gradient of the softmax function from vanishing due to excessively large dot product results. To query the dimensions of the vector and key vector; the division operation scales each element of the matrix.

[0069] Next, this invention introduces dialect relation graph attention to enhance cross-dialect information transmission. The dialect topology graph constructed in step S2 is used as a basis. And the probability of each frame's features belonging to each dialect. Construct a sequence-level dialect relation adjacency matrix :

[0070] ,

[0071] in: Adjacency matrix The Middle Line number The element of the column represents the first element. Frame and the The connection strength between frames based on dialect relationships; Total number of dialects; For the first Frame Dialect The probability of belonging to the group ranges from 0 to 1. For the first Frame Dialect The probability of belonging to the group ranges from 0 to 1. Dialects in the dialect topology map and dialects The edge weights between them come from the adjacency matrix constructed in step S2. Double summation symbol This indicates that the weighted sum is calculated by traversing all dialect pairs; the meaning of this formula is: the first... Frame and the The connection strength of a frame is equal to the expected value of the phonological similarity between the dialects to which they belong, and the dialect with a higher probability of belonging contributes more to the connection strength.

[0072] To incorporate dialect relationship information into attention calculation, a graph attention score matrix is ​​defined. :

[0073] ,

[0074] in: The graph attention score matrix has dimensions of . ; The previously calculated sequence-level dialect relation adjacency matrix has dimensions of . ; Let be a learnable edge embedding matrix with dimension . This matrix is ​​used to provide additional expressive power and is learned during training. This represents element-wise multiplication (Hadamard product), which means multiplying elements at corresponding positions; This is a layer normalization operation used to normalize the distribution of graph attention scores; the calculation formula is as follows: ,in The mean, For variance, To prevent division by zero, a small constant (usually taken as...) ).

[0075] The standard dot product attention score and the graph attention score are weighted and fused to obtain the final attention score matrix. :

[0076] ,

[0077] in: The fused attention score matrix has the following dimensions: ; The dot product attention score matrix has dimensions of . ; The graph attention score matrix has dimensions of . ; The fusion coefficient controls the relative importance of dot product attention and graph attention, with a value ranging from 0 to 1, and an optimal value of 0.6 to 0.8. It degenerates into standard attention when Only graph attention is used at this time; To assign weights to the graph attention, ensure that the sum of the weights of the two attention scores is 1; the addition operation adds the elements at corresponding positions in the two matrices.

[0078] Applying the softmax function to the fused attention score matrix yields the attention weight matrix. :

[0079] ,

[0080] in: This is the attention weight matrix, with dimension 1. Matrix elements Indicates the first The query position for the first Attention weights for each key position, ranging from 0 to 1, with the sum of weights for each row equal to 1; The softmax function is applied to each row of the matrix, and the calculation formula is as follows: ,in For the input vector, For the first One element; The fused attention score matrix has the following dimensions: .

[0081] Finally, the value matrix is ​​weighted and summed using the attention weight matrix to obtain the output features of cross-dialect attention fusion. :

[0082] ,

[0083] in: The output feature sequence of cross-dialect attention fusion has a dimension of ; This is the attention weight matrix, with dimension 1. ; It is a value matrix with dimension . Matrix multiplication The weighted summation is implemented, and the first feature is output. The row equals all rows of the value matrix according to the attention weights. The result of weighted summation.

[0084] Through the aforementioned cross-dialect attention fusion mechanism, the model can simultaneously consider the similarity of speech features (through dot product attention) and the phonetic relationships between dialects (through graph attention) when calculating attention. When processing the speech of a particular dialect, the model not only focuses on the feature dependencies within that dialect but can also draw on feature information from phonetically similar dialects, thereby achieving cross-dialect knowledge transfer and information sharing. This mechanism has a significant effect on improving the recognition accuracy of a few dialects and dialect boundary regions.

[0085] In multi-head attention mechanisms, the above computational processes are performed in parallel. Each time, different linear transformation weights are used. , , (in (representing the head index), to obtain Output features Then, this The output features are concatenated along the feature dimension and then subjected to a linear transformation to obtain the final multi-head attention output:

[0086] ,

[0087] in: For the final output of multi-head attention, the dimension is restored to... ; For the splicing operation, The outputs of the dimensions are concatenated on the last dimension, resulting in a concatenated dimension of... ; For the first The output of the size, with dimensions of ; The weight matrix for the linear transformation is a learnable parameter; matrix multiplication maps the concatenated features back to the original feature dimensions. .

[0088] After the multi-head attention output, the signal sequentially passes through residual connections, layer normalization, a feedforward neural network sub-layer, and then another set of residual connections and layer normalization, completing the processing of one encoder layer.

[0089] ,

[0090] in: This represents a feedforward neural network; The input features have a dimension of . ; The weight matrix is ​​the first-level linear transformation. The hidden layer dimension of a feedforward network is typically set to... ; This is the bias vector for the first layer; The ReLU activation function is applied to each element, and the calculation formula is as follows: ; This is the weight matrix for the second-level linear transformation; This is the bias vector for the second layer; the entire feedforward network implements nonlinear transformation and dimension recovery of the features.

[0091] Transformer encoder is composed of The encoder is composed of stacked layers, with the output of each layer serving as the input to the next layer, extracting higher-level feature representations layer by layer. In a preferred embodiment of the invention, the number of encoder layers is... Set to 6 to 12 layers, the number of heads for multi-head attention. The number of hidden layers in a feedforward network is set to 8 to 16. Set as feature dimension 4 times.

[0092] like Figure 4 As shown, in the processing of the Transformer encoder, this invention further introduces a dynamic dialect feature weighting mechanism as the third core innovation. The purpose of this mechanism is to dynamically adjust the feature weights according to the dialect confidence of the input speech, assigning higher weights to regions with significant dialect features and lower weights to regions with indistinct or confusing dialect features, thereby enhancing the model's sensitivity to dialect differences and preventing dialect features from being globally averaged or blurred.

[0093] The dialect feature dynamic weighting mechanism is implemented using a gating mechanism. For the output feature sequence of a certain layer of the encoder... First, calculate the dialect confidence vector for each frame. Then, use the dialect attribution probability obtained in step S2. This allows us to directly obtain the confidence distribution of each dialect in each frame. To quantify the salience of dialect features, we define a dialect feature salience value. :

[0094] ,

[0095] in: For the first The dialect feature saliency of a frame ranges from 0 to 1. The larger the value, the more significant the dialect feature of the frame. For the first The dialect affiliation probability vector of the frame, with dimension . ; The information entropy function is calculated using the following formula: ,in For dialect affiliation probability, The total number of dialects, Information entropy, calculated as the base-2 logarithm, measures the uncertainty of a probability distribution; when the probability of dialect affiliation is concentrated on a single dialect, the information entropy is relatively low. A larger entropy indicates significant dialectal features; when the probability of dialectal attribution is evenly distributed among multiple dialects, the information entropy is relatively large. Smaller indicates that dialectal features are not prominent or are confused.

[0096] Dynamic weight coefficients are generated using a gating network based on dialect feature saliency. :

[0097] ,

[0098] in: For the first The frame's gating weight vector, with dimension . Each element can take values ​​from 0 to 1; The Sigmoid activation function is applied to each element of the vector, and its calculation formula is: ; Let be the weight matrix of the gated network, and be the learnable parameters; This means concatenating three vectors into one vector, where For the first The feature vector of a frame (i.e.) The OK), This is a scalar of dialect feature saliency (extended to a 1-dimensional vector). This provides dialect distribution statistics for historical frames (calculated via a sliding window); the dimension of the concatenated vector is... ; is the bias vector of the gating network; the gating network learns comprehensive information from input features, dialect saliency, and historical dialect distribution, and outputs adaptive weight coefficients.

[0099] Dialect distribution statistics of historical frames Calculated using a sliding window:

[0100] ,

[0101] in: Dialects within the historical window The average probability of belonging; The value is the sliding window size, ranging from 5 to 20 frames, with a preferred value of 10 frames. For the frame index within the window; Ensure the window's initial position is not less than 1; Calculate dialects in all frames within the calculation window The sum of the probabilities of belonging; divided by The average value is obtained; this statistical information reflects the recent dialect distribution trend of speech, which helps the gating network make more stable weight decisions.

[0102] The generated gating weight vector is used to perform element-wise weighting of the features:

[0103] ,

[0104] in: For the weighted number of Frame feature vector, dimension is ; The gating weight vector has dimension 1. ; For the original number Frame feature vector, dimension is ; This represents element-wise multiplication, which means multiplying elements at corresponding positions. The weighting operation enables dynamic adjustment of features, strengthening dimensions with large gating weights and suppressing dimensions with small gating weights.

[0105] Apply the above weighting operation to all frames to obtain the weighted feature sequence. This feature sequence serves as the final output of the current encoder layer and is either passed to the next encoder layer or output to the decoder.

[0106] The advantages of the dialect feature dynamic weighting mechanism are as follows: First, it provides an objective measure of dialect feature strength by quantifying the saliency of dialect features through information entropy; second, the gating network comprehensively considers the features of the current frame, dialect saliency, and historical dialect distribution, and the generated weight coefficients have strong adaptability and robustness; third, the element-wise weighting operation preserves the fine-grained information of the features and avoids information loss caused by global averaging; fourth, this mechanism can be flexibly inserted into any layer of the encoder and work in conjunction with the cross-dialect attention fusion mechanism to further improve the model's ability to model dialect differences.

[0107] In practical implementation, the dynamic weighting mechanism for dialect features can be inserted into the middle or later layers of the encoder. Experiments show that applying this mechanism to layers 4 to 6 of the encoder yields the best results. At this point, the model has already extracted higher-level speech features, and the dialect information is fully represented. Dynamic weighting can effectively enhance the discriminability of dialect features.

[0108] go through The layer encoder processes the data and applies dynamic weighting of dialect features to key layers to ultimately obtain the encoded feature sequence. This sequence integrates speech content information and dialect feature information, and has strong expressive power and dialect sensitivity.

[0109] Step S4, Decoding and Text Output:

[0110] The encoded feature sequence output by the encoder The input is fed into the decoder for sequence-to-sequence decoding, generating the corresponding text sequence. The decoder employs a Transformer decoder architecture based on an attention mechanism. The decoder layer is composed of stacked layers. Each decoder layer contains three sub-layers: a masked multi-head self-attention sub-layer, an encoder-decoder multi-head cross-attention sub-layer, and a feedforward neural network sub-layer. Each sub-layer is followed by residual connections and layer normalization.

[0111] During the decoding process, the decoder generates text using an autoregressive approach, generating one token at a time and using the generated token sequence as input for the next decoding step. The decoder's input consists of two parts: first, the embedding representation of the target text (embedding of the ground truth labels during training and embedding of the generated tokens during inference); and second, the encoded feature sequence output by the encoder. .

[0112] The masked multi-head self-attention sublayer performs self-attention computation on the target sequence embedding input to the decoder. However, since decoding is autoregressive, the computation position... When representing, only positions 1 to 1 are considered. Information that cannot focus on future location The information at the end. This is achieved by applying a mask to the attention score matrix, setting the attention score for future positions to negative infinity, and then applying softmax to make the weights zero.

[0113] The encoder-decoder cross-attention sublayer implements the decoder's attention mechanism on the encoder's output. In this sublayer, the query vector... The key vector comes from the output of the previous sub-layer of the decoder. Sum value vector Output from encoder Through cross-attention, the decoder can focus on different positions of the input speech and dynamically extract relevant information from the encoded features based on the currently generated text content, thereby generating the next token.

[0114] The feedforward neural network sublayer performs a nonlinear transformation on the attention output, and its structure is the same as that of the feedforward network in the encoder.

[0115] The final output of the decoder passes through a linear layer and a softmax layer to obtain the probability distribution of each token in the vocabulary:

[0116] ,

[0117] in: For the first The vocabulary probability distribution vector for each decoding step has a dimension of . , For vocabulary size; For the last layer of the decoder in the first The output vector of the step has a dimension of ; The weight matrix for the output linear layer consists of learnable parameters. This is the bias vector for the output linear layer; The softmax function converts logits into a probability distribution, and the calculation formula is as follows: ,in For the input vector, For the first Each element.

[0118] During the inference phase, the decoder uses a beam search algorithm to generate text sequences, balancing generation quality and computational efficiency. The beam search maintains a matrix of size [size missing]. The set of candidate sequences, each decoding step expands from each candidate sequence. Given 10 possible next tokens, calculate the cumulative log probability of the expanded sequence and select the one with the highest cumulative log probability. Each sequence is used as a new candidate set, and this process is repeated until all candidate sequences have generated an end symbol or reached the maximum length. (Bundle width) The setting affects decoding quality and speed. In this invention, it is preferred to set it to 5 to 20. In most scenarios, setting it to 10 can achieve a better quality-speed balance.

[0119] Decoder layers Settings and number of encoder layers Related, usually set to or In a preferred embodiment of the invention, the number of encoder layers... Decoder layer number Bullish attention head count Feature Dimension Dimensions of hidden layers in feedforward networks vocabulary size (For Chinese character level) or (For Chinese sub-word level).

[0120] After processing by the decoder, the text sequence corresponding to the input speech is finally obtained, thus completing the multi-dialect speech recognition task.

[0121] The multi-dialect speech recognition model of this invention employs an end-to-end training approach, using a large-scale multi-dialect labeled dataset for training. The training dataset should contain as many Chinese dialect types as possible, with each dialect containing a sufficient number of speech-text pairs to ensure that the model can learn the speech features and language models of each dialect. The dataset size is preferably 500 to 5000 hours, covering 10 to 30 dialects.

[0122] The training employs a teacher-forcing strategy, meaning that during the training phase, the decoder's input uses real target text sequences instead of tokens generated by the model itself. The loss function is cross-entropy loss, which calculates the cross-entropy between the probability distribution of the decoder's output and the true labels.

[0123] ,

[0124] in: The cross-entropy loss function; The length of the target text sequence; For the first The real label of each location (index of the real token); Indicates position 1 to The actual label sequence; The input is a sequence of speech features; For the model to predict the first Each position is a real label. The probability, derived from the probability distribution vector output by the decoder. The One element; It is the natural logarithm; the negative sign and averaging operation constitute the standard form of cross-entropy loss.

[0125] The optimizer uses the Adam optimizer, and the learning rate is dynamically adjusted using a warmup strategy.

[0126] ,

[0127] in: For the first The learning rate for each training step; The base learning rate is preferably set to to ; This is the current training step count, starting from 1; The number of preheating steps is preferably set to 4,000 to 10,000 steps; Take the smaller of the two values; This is the learning rate decay term, which gradually reduces the learning rate after the warm-up phase; As a warm-up term, the learning rate increases linearly from 0 to its peak during the warm-up phase. This learning rate scheduling strategy gradually increases the learning rate in the early stages of training to stabilize training, and gradually decreases the learning rate in the later stages of training to refine the parameters.

[0128] The training process employs gradient accumulation and mixed-precision training techniques to improve training efficiency. Gradient accumulation sums the gradients of multiple mini-batches before updating the parameters, which is equivalent to increasing the batch size without increasing GPU memory consumption. Mixed-precision training uses 16-bit floating-point numbers (FP16) for forward and backward propagation and 32-bit floating-point numbers (FP32) for storing master parameters and updating parameters, significantly reducing GPU memory usage and computation time while maintaining training stability.

[0129] During training, the batch size is set to 16 to 64 samples (adjusted according to GPU memory capacity), the gradient accumulation step is 2 to 8 steps, and the total equivalent batch size is 32 to 512 samples. The number of training epochs is set to 50 to 200 epochs, and early stopping is performed based on the word error rate on the validation set. Training stops when the validation set performance no longer improves for 10 consecutive epochs.

[0130] To prevent overfitting, several regularization techniques were employed during training: Dropout was applied to each layer of the encoder and decoder, with Dropout probabilities set to 0.1 to 0.3; Label Smoothing was used, with a smoothing coefficient set to 0.1; and L2 regularization was applied to the model parameters, with a weight decay coefficient set to [missing value]. to .

[0131] After training, the model performance was evaluated on independent multi-dialect test sets. Evaluation metrics included character error rate (CER), word error rate (WER), and individual recognition accuracy for each dialect. Experimental results show that the method of this invention reduces the character error rate by 18% to 25% compared to the baseline model (standard Transformer speech recognition model) in multi-dialect scenarios, and improves the recognition accuracy for a few dialects by 30% to 40%, fully validating the effectiveness of dialect topology space mapping, cross-dialect attention fusion, and dynamic weighting mechanism of dialect features.

[0132] To more clearly illustrate the implementation effect of this invention, a specific application scenario example is given below. Assume the input is a 3-second audio clip from a dialect-intersection region in southern China, which mixes features of Cantonese and Hakka, and the content is "Today's weather is beautiful." After 16kHz resampling and preprocessing, the audio is divided into 300 frames. A 128-dimensional log-Mel filter bank feature is extracted from each frame to form the original acoustic feature sequence. .

[0133] In the dialect topological space mapping stage, the feature sequence undergoes a linear transformation to map to a 512-dimensional manifold tangent space, resulting in... For each frame, the model calculates the probability of its belonging to 10 dialects (including Cantonese, Hakka, Minnan, etc.). In this example, the average probability of belonging to Cantonese for the first 150 frames is 0.7, and the average probability of belonging to Hakka for the last 150 frames is 0.6, reflecting the dialect mixture characteristics of the speech. Based on the dialect belonging probabilities and a pre-constructed dialect topology map (where the edge weight between Cantonese and Hakka is 0.75, indicating a high phonetic similarity), the model constructs a sequence-level adjacency matrix to provide topological structure information for subsequent cross-dialect attention fusion.

[0134] In the cross-dialect attention fusion stage of the encoder, the model calculates dot-product attention and graph attention, and performs weighted fusion with a fusion coefficient The cross-dialect attention mechanism enables the Cantonese features in the first half to focus on the parts of the Hakka features in the second half that are similar to Cantonese, and vice versa, thus achieving cross-dialect information transfer. In the 6th layer of the encoder, a dialect feature dynamic weighting mechanism is applied. The model calculates the dialect feature saliency of each frame and finds that the dialect feature saliency of the 50th to 100th frames and the 200th to 250th frames is relatively high (low information entropy, concentrated belonging probabilities), corresponding to the typical syllables of Cantonese and Hakka respectively, and these frames are given higher gating weights (average of 0.9); while the dialect feature saliency of the 140th to 160th frames is relatively low (high information entropy, dispersed belonging probabilities), corresponding to the transitional area at the dialect boundary, and these frames are given lower gating weights (average of 0.4). Through dynamic weighting, the model strengthens the areas with significant dialect features, suppresses the areas with dialect confusion, and improves the distinguishability of the features.

[0135] After being processed by the 12-layer encoder, a coded feature sequence is obtained. The decoder uses a beam search algorithm (beam width of 10) for decoding and gradually generates the character sequence. When generating the character "今", the attention of the decoder mainly focuses on the Cantonese pronunciation features of the first 50 frames; when generating the character "日", the attention shifts to the 50th to 80th frames; when generating the characters "天", "气", "好", the attention successively focuses on the corresponding speech segments; when generating the character "靓" (a Cantonese term), the attention focuses on the frames with Cantonese features in the second half. Finally, the model correctly outputs the text sequence "今日天气好靓", achieving accurate recognition of the dialect-mixed speech.

[0136] This example demonstrates the advantages of the method of the present invention in processing speech in dialect border areas: accurately capturing dialect mixing features through dialect topological space mapping, achieving effective interaction of Cantonese and Hakka features through cross-dialect attention fusion, strengthening dialect significant regions and suppressing confusing regions through dynamic weighting of dialect features, and finally achieving high-precision speech recognition. Compared with the standard Transformer model (without a dialect modeling mechanism), the method of the present invention avoids misidentifying the word "liang" as the Mandarin words "beautiful" or "very" in this example, reflecting an accurate grasp of dialect-specific vocabulary.

[0137] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A multi-dialect speech recognition method based on Transformer self-attention mechanism, characterized in that, Includes the following steps: Step S1: Obtain a multi-dialect audio dataset, preprocess and extract features from the audio dataset to obtain the original acoustic feature sequence; Step S2: The original acoustic feature sequence is processed using the dialect topology space mapping module to construct a topology space representation of the dialect features and obtain a topology mapping feature sequence. Step S3: Input the topology mapping feature sequence into the Transformer encoder. In the Transformer encoder, the attention weights are calculated using a cross-dialect attention fusion mechanism, and the features are weighted using a dialect feature dynamic weighting mechanism to obtain the encoded feature sequence. Step S4: Input the encoded feature sequence into the decoder for decoding to obtain the corresponding text sequence.

2. The multi-dialect speech recognition method based on Transformer self-attention mechanism according to claim 1, characterized in that, In step S2, the dialect topology space mapping module adopts a dialect feature mapping method based on Riemannian manifolds to map the acoustic features of different dialects into a unified topology space, and calculates the similarity between dialects by geodesic distance.

3. The multi-dialect speech recognition method based on Transformer self-attention mechanism according to claim 2, characterized in that, The construction process of the dialect topology space mapping module includes: First, a dialect classifier is trained using dialect-annotated data to extract prototype feature vectors of different dialects; Then, a dialect topology graph is constructed in the feature space, where nodes represent dialect prototypes and edge weights represent the phonetic distance between dialects. Finally, the manifold structure is defined using the Riemannian metric tensor, and the acoustic features are mapped to the manifold space.

4. The multi-dialect speech recognition method based on Transformer self-attention mechanism according to claim 3, characterized in that, The construction of the dialect topology graph includes: Calculate the cosine similarity between dialect prototype feature vectors; Based on the cosine similarity, the similarity is mapped to edge weights using a Gaussian kernel function, and dialects with similar phonology are connected by edges with larger weights. Construct an adjacency matrix, where the elements of the adjacency matrix represent the connection weights between dialects.

5. The multi-dialect speech recognition method based on Transformer self-attention mechanism according to claim 3, characterized in that, The Riemannian manifold mapping includes: The original acoustic features are mapped to the manifold tangent space through a linear transformation; Calculate the similarity between the features of each frame and the prototypes of each dialect to obtain the dialect attribution probability distribution; Based on the dialect affiliation probability, the basic metric matrices corresponding to different dialects are weighted and summed to construct the Riemann metric tensor. The geodesic distance between eigenvectors on the manifold is calculated using the Riemannian metric tensor.

6. The multi-dialect speech recognition method based on Transformer self-attention mechanism according to claim 1, characterized in that, In step S3, the calculation process of the cross-dialect attention fusion mechanism includes: A linear transformation is performed on the input topological mapping features to obtain the query matrix, key matrix, and value matrix; A sequence-level dialect adjacency matrix is ​​constructed based on a dialect relation graph, and the adjacency matrix encodes the topological adjacency information between dialects; Calculate the dot product attention score matrix and the graph attention score matrix; The dot product attention score matrix and the graph attention score matrix are weighted and fused to obtain a fused attention score matrix; Applying the softmax function to the fused attention score matrix yields the attention weight matrix; The value matrix is ​​weighted and summed using the attention weight matrix to obtain the feature representation of cross-dialect fusion.

7. The multi-dialect speech recognition method based on Transformer self-attention mechanism according to claim 6, characterized in that, The construction of the sequence-level dialect relation adjacency matrix includes: For any two frames in the sequence, iterate through all dialect pairs and calculate the product of the probability of each frame belonging to each dialect and the edge weight between dialects. Summing the products of all dialect pairs yields the connection strength between two frames based on dialect relationships; The connection strength is used as the corresponding element of the sequence-level dialect relation adjacency matrix.

8. The multi-dialect speech recognition method based on Transformer self-attention mechanism according to claim 6, characterized in that, The calculation of the graph attention score matrix includes: Construct a learnable edge embedding matrix; Perform element-wise multiplication between the sequence-level dialect relation adjacency matrix and the edge embedding matrix; The multiplication result is layer-normalized to obtain the graph attention score matrix; The weighted fusion uses a fusion coefficient to control the relative importance of dot product attention and graph attention, and the value of the fusion coefficient ranges from 0.6 to 0.

8.

9. The multi-dialect speech recognition method based on Transformer self-attention mechanism according to claim 1, characterized in that, In step S3, the dialect feature dynamic weighting mechanism includes: The dialect feature saliency of each frame is calculated using the dialect attribution probability, and the dialect feature saliency is quantized by the inverse of information entropy; Based on dialect feature saliency, current frame features, and historical frame dialect distribution statistics, dynamic weight coefficients are generated through a gating network. The dynamic weighting coefficients are used to weight the features element by element, giving higher weights to regions with significant dialect features and lower weights to regions with indistinct or confusing dialect features.

10. The multi-dialect speech recognition method based on Transformer self-attention mechanism according to claim 9, characterized in that, The input to the gating network includes the feature vector of the current frame, the saliency of the dialect features of the current frame, and the dialect distribution statistics of historical frames calculated by a sliding window. The gating network outputs a gating weight vector through a sigmoid activation function. Each element of the gating weight vector has a value ranging from 0 to 1, which is used to adjust the corresponding dimension of the feature vector.

Images