A method and system for training a mixed lin-gao and mandarin speech recognition model

Abstract

The application relates to the technical field of speech recognition, and discloses a training method of a Lin'gao-Chinese mixed speech recognition model, an end-to-end mixed speech recognition model constructed based on a Conformer model, and the method comprises the following steps: inputting a mixed speech signal containing Lin'gao and Chinese, and extracting an acoustic feature sequence of the mixed speech signal; inputting the feature sequence into a preset encoder, performing down-sampling to reduce feature redundant information, and then performing encoding on the encoder to generate a hidden vector; a decoder decodes the vector, searches frame by frame in a decoding graph according to the weight on a transfer arc, and calculates the weight score in each path; the path with the highest score is selected as an optimal path, and state information is continuously updated; and a label sequence corresponding to the optimal path is output as a final recognition result. The application enhances the ability of the model to extract local subtle features, improves the language modeling capability of the model, and improves the recognition accuracy of the model.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a method and system for training a mixed speech recognition model of Lingao dialect and Mandarin. Background Technology

[0002] Chinese dialects not only represent a form of written symbolism but also embody profound Chinese culture. Dialects also possess rich regional cultural characteristics and social customs. Today, marginalized dialects are facing the impact of globalization and are in danger of gradually disappearing. Therefore, research on dialect speech recognition, which constitutes a core aspect of dialect preservation, has significant practical implications.

[0003] The core goal of speech recognition technology is to convert sound into text, enabling machines to understand what users are saying during voice communication. The advent of the big data era has generated massive amounts of voice data, greatly promoting the development of speech recognition technology. Traditional speech recognition technologies are represented by Gaussian mixture models and hidden Markov models. However, because Gaussian mixture models do not utilize the contextual information of frames, cannot learn deep nonlinear feature transformations, and their solution algorithms are prone to getting trapped in local extrema, their actual recognition performance is not ideal.

[0004] With technological advancements, traditional speech recognition models can no longer meet the growing demands. Deep learning technology, with its significant advantages in processing large amounts of data, has been applied to speech recognition, leading to rapid advancements and higher accuracy. The continuous development of end-to-end speech recognition technology has greatly reduced the complexity of speech recognition model structures, integrating the entire recognition process into a single network that directly maps the input audio to a text sequence. However, to improve accuracy, the number of neural network layers continues to increase, resulting in increasingly complex network structures. This leads to excessively large model parameters, high computational complexity, and high latency, making it difficult to meet practical application needs. Summary of the Invention

[0005] This invention provides a training method and system for a mixed Lingao dialect and Mandarin speech recognition model. The model structure adopts a preset encoder and a CTC WFST search decoder, which enhances the model's ability to extract local subtle features, improves the model's language modeling ability, and increases the model's recognition accuracy.

[0006] This invention provides a training method for a mixed speech recognition model of Lingao dialect and Mandarin, based on an end-to-end mixed speech recognition model constructed using a Conformer model. The end-to-end mixed speech recognition model has an encoder-decoder structure, where the encoder consists of a preset encoder containing multiple identical Conformer modules. The decoder employs a CTC WFST search decoder composed of a CTC decoder and an N-gram language model, where the N-gram language model is represented as a weighted finite state converter. The training method for the end-to-end mixed speech recognition model includes:

[0007] Input a mixed speech signal containing Lingao dialect and Mandarin, and extract the acoustic feature sequence S of the mixed speech signal;

[0008] The feature sequence S is input into a preset encoder, the feature sequence S is downsampled to reduce feature redundancy information, and then encoded by the encoder to generate a hidden vector O;

[0009] The CTC WFST search decoder decodes vector O and uses the CTC Prefix Beam Search algorithm to search frame by frame in the decoded graph based on the weights on the transition arcs, calculating the weight score in each path.

[0010] Select the path with the highest score as the optimal path and continuously update the status information;

[0011] The label sequence corresponding to the optimal path is output as the final recognition result Y.

[0012] Furthermore, the preset encoder includes a speech enhancement module, a convolutional downsampling module, a linear layer, and a Conformer module; the Conformer module includes a feedforward neural network module, a multi-head attention module, and a convolutional module, with the convolutional module and the multi-head attention module sandwiched between two half-step feedforward neural network modules, and residual structures are used in the feedforward neural network module, the convolutional module, and the multi-head attention module.

[0013] In the preset encoder, x n y represents the input of the nth Conformer module. n This indicates its corresponding output. The calculation process of the Conformer module is as follows:

[0014]

[0015] x MHA =x F1 +MHA(x F1 )

[0016] xConv =x MHA +Conv(x MHA )

[0017]

[0018] Wherein, FFN represents the computation function of the feedforward neural network module, MHA represents the computation function of the multi-head attention module, Conv represents the computation function of the convolution module, and LayerNorm represents the normalization function.

[0019] Furthermore, the convolutional module consists of Swish activation, GLU activation layers, pointwise convolution, depthwise convolution, batch normalization, and normalization layers; the GLU activation function determines which information can be passed to the next module, and its formula is:

[0020]

[0021] Where X represents the input, W and V represent different convolution kernels, and b and c are the offsets;

[0022] The normalization layer integrates information, repositions and optimizes network parameters, and avoids differences in input distribution by dynamically adjusting the length of the input vector. The linear layer connects the neurons in each layer of the neural network to the neurons in the previous layer, realizing linear transformation operations.

[0023] Furthermore, the feedforward neural network module consists of two internal linear layers and uses the Swish activation function for linear transformation, while employing Dropout to reduce overfitting.

[0024] The feedforward neural network is used to update each state information of the attention layer output vector, and its formula is as follows:

[0025] FFN(x) = max(0, xW1+b1)W2+b2

[0026] Where W represents the weight, b represents the bias, and x represents the input.

[0027] Furthermore, the CTC WFST search decoder includes two operations during decoding: constructing a decoding graph and decoding.

[0028] The construction of the decoding graph combines the information of each part of T, L, and G, where the modeling unit is represented by T, the dictionary by L, and the language model by G. Chinese characters are used as the modeling unit T in the end-to-end hybrid speech recognition model. Words or sentences are split into modeling units and combined to form the dictionary L. The N-gram language model is converted into the WFST representation to form G, where WFST is a decoding form commonly used in the field of speech recognition, which can represent the N-gram language model in the form of a graph.

[0029] Furthermore, it also includes a multi-head attention decoder, the output of which is connected to the CTC decoder and the multi-head attention decoder. The multi-head attention decoder consists of multiple Transformer-structured decoders, including a multi-head attention layer and a masked multi-head attention layer.

[0030] Furthermore, the multi-head attention decoder decodes in an autoregressive manner. The label output by the decoder at the previous time step is input to the masked multi-head attention layer after passing through the output embedding layer and position encoding, and the corresponding Q, K and V values ​​are generated. Part of the input to the multi-head attention layer is the output K vector and V vector of the encoder, and the other part is the output Q vector of the masked multi-head attention layer. Finally, the complete recognition result is output through the Softmax layer.

[0031] The position encoding in the multi-head attention decoder uses sine and cosine functions of different frequencies, and its formula is as follows:

[0032]

[0033]

[0034] Where pos refers to the position of the word in the sentence, i represents the dimension of the embedded word vector, 2i represents an even dimension, 2i+1 represents an odd dimension, each dimension of the positional encoding corresponds to a sine signal, and the corresponding wavelength increases geometrically from 2π to 10000·2π.

[0035] This invention also provides a training system for a mixed speech recognition model of Lingao dialect and Mandarin, based on an end-to-end mixed speech recognition model constructed using a Conformer model. The end-to-end mixed speech recognition model has an encoder-decoder structure, where the encoder consists of a preset encoder containing multiple identical Conformer modules. The decoder employs a CTC WFST search decoder composed of a CTC decoder and an N-gram language model, where the N-gram language model is represented as a weighted finite state converter. The training system for the end-to-end mixed speech recognition model includes:

[0036] The input module is used to input a mixed speech signal containing Lingao dialect and Mandarin, and to extract the acoustic feature sequence S of the mixed speech signal;

[0037] The encoding module is used to input the feature sequence S into a preset encoder, downsample the feature sequence S to reduce feature redundancy information, and then encode it to generate a hidden vector O.

[0038] The decoding module is used to decode vector O by the CTC WFST search decoder. The CTC PrefixBeam Search algorithm searches frame by frame in the decoded graph based on the weights on the transition arcs and calculates the weight score in each path.

[0039] The selection module is used to select the path with the highest score as the optimal path and continuously update the status information.

[0040] The output module is used to output the label sequence corresponding to the optimal path as the final recognition result Y.

[0041] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0042] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0043] The beneficial effects of this invention are as follows:

[0044] 1. This invention combines the Conformer model and the N-gram language model, proposing a model structure employing a pre-defined encoder and a CTCWFSTsearch decoder. The CTCWFST search decoder consists of an N-gram language model and a CTC decoder. This not only enhances the model's ability to extract subtle local features but also improves its language modeling capabilities, further increasing the model's recognition accuracy. Data augmentation methods are used to mitigate the impact of insufficient corpus data, enhancing the model's robustness.

[0045] 2. Based on the Conformer model, a method is proposed that uses a CTC decoder and a multi-head attention mechanism for joint decoding. This method is based on a pre-defined encoder and uses a CTC decoder and a multi-head attention decoder for joint decoding. The probabilities of the partial outputs of the two decoders are weighted and summed, and the result with the highest probability is used as the final model output. Combined with the N-gram language model, this further improves the model's recognition accuracy. Attached Figure Description

[0046] Figure 1This is a schematic diagram of a method flow according to an embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram of the end-to-end hybrid speech recognition model in this invention.

[0048] Figure 3 This is a schematic diagram of the encoder structure in this invention.

[0049] Figure 4 This is a schematic diagram of the device structure according to an embodiment of the present invention.

[0050] Figure 5 This is a schematic diagram of the internal structure of a computer device according to an embodiment of the present invention.

[0051] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0053] This invention provides a training method for a mixed speech recognition model of Lingao dialect and Mandarin, and an end-to-end mixed speech recognition model built based on the Conformer model, as follows: Figure 2 As shown, the end-to-end hybrid speech recognition model is an encoder-decoder structure. The encoder is composed of a preset encoder, which contains multiple identical Conformer modules. The decoder is a CTC WFSTsearch decoder composed of a CTC decoder and an N-gram language model. The N-gram language model is represented in the form of a weighted finite-state transducer (WFST).

[0054] like Figure 1 As shown, the training method for the end-to-end hybrid speech recognition model includes:

[0055] S1. Input a mixed speech signal containing Lingao dialect and Mandarin, and extract the acoustic feature sequence S of the mixed speech signal;

[0056] S2. Input the feature sequence S into a preset encoder, downsample the feature sequence S to reduce feature redundancy information, and then encode it to generate a hidden vector 0.

[0057] S3. The CTC WFST search decoder decodes vector 0 and uses the CTC Prefix Beam Search algorithm to search frame by frame in the decoded graph based on the weights on the transition arcs, calculating the weight score in each path.

[0058] S4. Select the path with the highest score as the optimal path and continuously update the status information;

[0059] S5. Output the label sequence corresponding to the optimal path as the final recognition result Y.

[0060] Encoder structure:

[0061] like Figure 3 As shown, the Encoder part uses a preset encoder. The right side shows the structure of the preset encoder, and the left side shows the structure of the Conformer module. The preset encoder includes a speech enhancement module, a convolutional downsampling module, linear layers, and a Conformer module. The Conformer module includes a feedforward neural network module, a multi-head attention module, and a convolutional module. The convolutional module and the multi-head attention module are sandwiched between two half-step feedforward neural network modules. Residual structures are used in the feedforward neural network module, the convolutional module, and the multi-head attention module. This effectively improves the shortcomings of the Transformer model and further enhances the overall performance of the model.

[0062] In the preset encoder, x n y represents the input of the nth Conformer module. n This indicates its corresponding output. The calculation process of the Conformer module is as follows:

[0063]

[0064] x MHA =x F1 +MHA(x F1 )

[0065] x Conv =x MHA +Conv(x MHA )

[0066]

[0067] Wherein, FFN represents the computation function of the feedforward neural network module, MHA represents the computation function of the multi-head attention module, Conv represents the computation function of the convolution module, and LayerNorm represents the normalization function.

[0068] The multi-head attention module employs a residual structure to prevent the loss of important information. Simultaneously, Dropout technology is used to temporarily discard neural network units from the model with a certain probability, effectively reducing overfitting and further aiding in training deeper network layers.

[0069] Scaled dot product attention is used in multi-head attention layers, and its calculation method is as follows:

[0070]

[0071] Where Z represents the attention output, Q, K, and V are derived from the input feature sequences through different linear transformations, Q represents the query value, K represents the key value, and V represents the value, with Q and K having a dimension of d. k The dimension of V is d v . This represents the scaling factor, effectively avoiding the problem of gradient optimization being difficult due to excessively large Softmax values.

[0072] The calculation of the multi-head attention mechanism is as follows:

[0073] MultiHead(Q,K,V)=Concat(head1,...,head n W O

[0074]

[0075] First, the input feature sequence is mapped to multiple sets of Q, K, and V, and attention is calculated for each set. Then, the outputs of each attention mechanism are integrated to form a complete output. The head is one such set. i This represents the attention head of the i-th group. W represents the weight matrix corresponding to each group of Q, K, and V. O This represents the additional weight matrix.

[0076] The convolutional module consists of Swish activation, GLU activation layers, pointwise convolution, depthwise convolution, BatchNorm, and normalization layers. It utilizes a residual structure to effectively enhance gradient propagation and prevent gradient vanishing. Depthwise separable convolution is employed, dividing ordinary convolution operations into pointwise and depthwise convolutions. This reduces computational cost, allowing more feature information to be learned with fewer parameters.

[0077] The GLU activation function determines which information can be passed to the next module, and its formula is as follows:

[0078]

[0079] Where X represents the input, W and V represent different convolution kernels, and b and c are the offsets;

[0080] The normalization layer integrates information, repositions and optimizes network parameters, and avoids differences in input distribution by dynamically adjusting the length of the input vector. The linear layer connects the neurons in each layer of the neural network to the neurons in the previous layer, realizing linear transformation operations.

[0081] The feedforward neural network module consists of two internal linear layers and uses the Swish activation function for linear transformation, while employing Dropout to reduce overfitting.

[0082] The feedforward neural network is used to update each state information of the attention layer output vector, and its formula is as follows:

[0083] FFN(x) = max(0, xW1+b1)W2+b2

[0084] Where W represents the weight, b represents the bias, and x represents the input.

[0085] During training, the use of Specaugment data augmentation effectively transforms overfitting into underfitting, thus improving speech recognition performance. Furthermore, the convolutional downsampling module effectively reduces computational load while maintaining the encoder's expressive power and the model's recognition accuracy.

[0086] Decoder structure:

[0087] In the decoder section, a CTC WFST search decoder, composed of a CTC decoder and an N-gram language model, is used. The CTC WFST search decoder includes two operations during decoding: constructing the decoding graph and decoding.

[0088] The construction of the decoding graph combines the information of each part of T, L, and G, where the modeling unit is represented by T, the dictionary by L, and the language model by G. Chinese characters are used as the modeling unit T in the end-to-end hybrid speech recognition model. Words or sentences are split into modeling units and combined to form the dictionary L. The N-gram language model is converted into the WFST representation to form G, where WFST is a decoding form commonly used in the field of speech recognition, which can represent the N-gram language model in the form of a graph.

[0089] Language models are an indispensable part of traditional speech recognition systems, primarily calculating the probability of word sequences occurring. Applied to end-to-end speech recognition models, they can compensate for the limitations of end-to-end models in language modeling, providing more accurate recognition results. N-gram language models, used to represent the probability of combinations between n words, are still widely used in speech recognition and have mature training tools and methods.

[0090] Multi-head attention decoder:

[0091] The output of the preset encoder is connected to the CTC decoder and the multi-head attention decoder. The multi-head attention decoder consists of multiple Transformer-structured decoders, including a multi-head attention layer and a masked multi-head attention layer.

[0092] The multi-head attention decoder decodes in an autoregressive manner. The label output by the decoder in the previous time step is input to the masked multi-head attention layer after passing through the output embedding layer and position encoding, and the corresponding Q, K and V values ​​are generated. Part of the input to the multi-head attention layer is the output K vector and V vector of the encoder, and the other part is the output Q vector of the masked multi-head attention layer. Finally, the complete recognition result is output through the Softmax layer.

[0093] The position encoding in the multi-head attention decoder uses sine and cosine functions of different frequencies, and its formula is as follows:

[0094]

[0095]

[0096] Where pos refers to the position of the word in the sentence, i represents the dimension of the embedded word vector, 2i represents an even dimension, 2i+1 represents an odd dimension, each dimension of the positional encoding corresponds to a sine signal, and the corresponding wavelength increases geometrically from 2π to 10000·2π.

[0097] like Figure 4 As shown, this invention also provides a training system for a mixed speech recognition model of Lingao dialect and Mandarin, based on an end-to-end mixed speech recognition model constructed using a Conformer model. The end-to-end mixed speech recognition model has an encoder-decoder structure, where the encoder consists of a preset encoder containing multiple identical Conformer modules. The decoder uses a CTC WFST search decoder composed of a CTC decoder and an N-gram language model, where the N-gram language model is represented as a weighted finite state converter. The training system for the end-to-end mixed speech recognition model includes:

[0098] Input module 1 is used to input a mixed speech signal containing Lingao dialect and Mandarin, and to extract the acoustic feature sequence S of the mixed speech signal;

[0099] Encoding module 2 is used to input the feature sequence S into a preset encoder, downsample the feature sequence S to reduce feature redundancy information, and then encode it to generate a hidden vector O.

[0100] Decoding module 3 is used by the CTC WFST search decoder to decode vector O, and uses the CTC PrefixBeam Search algorithm to search frame by frame in the decoded graph according to the weights on the transition arcs, and calculates the weight score in each path;

[0101] Module 4 is used to select the path with the highest score as the optimal path and continuously update the status information.

[0102] Output module 5 is used to output the label sequence corresponding to the optimal path as the final recognition result Y.

[0103] In one embodiment, the preset encoder includes a speech enhancement module, a convolutional downsampling module, a linear layer, and a Conformer module; the Conformer module includes a feedforward neural network module, a multi-head attention module, and a convolutional module, with the convolutional module and the multi-head attention module sandwiched between two half-step feedforward neural network modules, and residual structures are used in the feedforward neural network module, the convolutional module, and the multi-head attention module.

[0104] In the preset encoder, x n y represents the input of the nth Conformer module. n This indicates its corresponding output. The calculation process of the Conformer module is as follows:

[0105]

[0106] x MHA =x F1 +MHA(x F1 )

[0107] x Conv =x MHA +Conv(x MHA )

[0108]

[0109] Wherein, FFN represents the computation function of the feedforward neural network module, MHA represents the computation function of the multi-head attention module, Conv represents the computation function of the convolution module, and LayerNorm represents the normalization function.

[0110] In one embodiment, the convolutional module consists of Swish activation, GLU activation layers, pointwise convolution, depthwise convolution, batch normalization, and normalization layers; the GLU activation function determines which information can be passed to the next module, and its formula is:

[0111]

[0112] Where X represents the input, W and V represent different convolution kernels, and b and c are the offsets;

[0113] The normalization layer integrates information, repositions and optimizes network parameters, and avoids differences in input distribution by dynamically adjusting the length of the input vector. The linear layer connects the neurons in each layer of the neural network to the neurons in the previous layer, realizing linear transformation operations.

[0114] In one embodiment, the feedforward neural network module consists of two internal linear layers and uses the Swish activation function for linear transformation, while employing Dropout to reduce overfitting.

[0115] The feedforward neural network is used to update each state information of the attention layer output vector, and its formula is as follows:

[0116] FFN(x) = max(0, xW1+b1)W2+b2

[0117] Where W represents the weight, b represents the bias, and x represents the input.

[0118] In one embodiment, the CTC WFST search decoder includes two operations during decoding: constructing a decoding graph and decoding.

[0119] The construction of the decoding graph combines the information of each part of T, L, and G, where the modeling unit is represented by T, the dictionary by L, and the language model by G. Chinese characters are used as the modeling unit T in the end-to-end hybrid speech recognition model. Words or sentences are split into modeling units and combined to form the dictionary L. The N-gram language model is converted into the WFST representation to form G, where WFST is a decoding form commonly used in the field of speech recognition, which can represent the N-gram language model in the form of a graph.

[0120] In one embodiment, a multi-head attention decoder is also included, wherein the output of the preset encoder is connected to the CTC decoder and the multi-head attention decoder, and the multi-head attention decoder is composed of multiple Transformer-structured decoders, including a multi-head attention layer and a masked multi-head attention layer.

[0121] In one embodiment, the multi-head attention decoder decodes in an autoregressive manner. The label output by the decoder at the previous time step is input to the masked multi-head attention layer after passing through the output embedding layer and position encoding, and the corresponding Q, K and V values ​​are generated. Part of the input to the multi-head attention layer is the output K vector and V vector of the encoder, and the other part is the output Q vector of the masked multi-head attention layer. Finally, the complete recognition result is output through the Softmax layer.

[0122] The position encoding in the multi-head attention decoder uses sine and cosine functions of different frequencies, and its formula is as follows:

[0123]

[0124]

[0125] Where pos refers to the position of the word in the sentence, i represents the dimension of the embedded word vector, 2i represents an even dimension, 2i+1 represents an odd dimension, each dimension of the positional encoding corresponds to a sine signal, and the corresponding wavelength increases geometrically from 2π to 10000·2π.

[0126] The modules described above are used to perform the respective steps in the above-mentioned training method for the mixed speech recognition model of Lingao dialect and Mandarin. The specific implementation methods are as described in the above-mentioned method embodiments, and will not be repeated here.

[0127] like Figure 5 As shown, the present invention also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores all data required for the training process of the mixed Lingao and Mandarin speech recognition model. The network interface is used for communication with external terminals via a network connection. The computer program is executed by the processor to implement the training method for the mixed Lingao and Mandarin speech recognition model.

[0128] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment on which the present application is applied.

[0129] An embodiment of this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the above-described methods for training a mixed speech recognition model of Lingao dialect and Mandarin.

[0130] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0131] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0132] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for training a mixed speech recognition model of Lingao dialect and Mandarin, characterized in that, An end-to-end hybrid speech recognition model is constructed based on the Conformer model. The end-to-end hybrid speech recognition model is an encoder-decoder structure. The encoder is composed of a preset encoder, which contains multiple identical Conformer modules. The decoder adopts a CTC WFST search decoder composed of a CTC decoder and an N-gram language model. The N-gram language model is represented in the form of a weighted finite state converter. The preset encoder includes a speech enhancement module, a convolutional downsampling module, a linear layer, and a Conformer module; the Conformer module includes a feedforward neural network module, a multi-head attention module, and a convolutional module. The convolutional module and the multi-head attention module are sandwiched between two half-step feedforward neural network modules. Residual structures are used in the feedforward neural network module, the convolutional module, and the multi-head attention module. In the preset encoder, using This represents the input of the nth Conformer module. This indicates the corresponding output. The calculation process of the Conformer module is as follows: Wherein, FFN represents the computation function of the feedforward neural network module, MHA represents the computation function of the multi-head attention module, Conv represents the computation function of the convolution module, and LayerNorm represents the normalization function. The feedforward neural network module consists of two internal linear layers and uses the Swish activation function for linear transformation, while employing Dropout to reduce overfitting. The feedforward neural network is used to update each state information of the attention layer output vector, and its formula is as follows: Where W represents the weight, b represents the bias, and x represents the input; The CTC WFST search decoder includes two operations during decoding: constructing a decoding graph and decoding. The construction of the decoding graph combines the information of each part of T, L, and G, where the modeling unit is represented by T, the dictionary by L, and the language model by G. Chinese characters are used as the modeling unit T in the end-to-end hybrid speech recognition model. Words or sentences are split into modeling units and combined to form the dictionary L. The N-gram language model is converted into the WFST representation to form G, where WFST is a decoding form commonly used in the field of speech recognition, which can represent the N-gram language model in the form of a graph. The training method for the end-to-end hybrid speech recognition model includes: Input a mixed speech signal containing Lingao dialect and Mandarin, and extract the acoustic feature sequence S of the mixed speech signal; The feature sequence S is input into a preset encoder, the feature sequence S is downsampled to reduce feature redundancy information, and then encoded by the encoder to generate a hidden vector O; The CTC WFST search decoder decodes vector O and uses the CTC Prefix Beam Search algorithm to search frame by frame in the decoded graph based on the weights on the transition arcs, calculating the weight score in each path. Select the path with the highest score as the optimal path and continuously update the status information; The label sequence corresponding to the optimal path is output as the final recognition result Y.

2. The method for training a mixed speech recognition model of Lingao dialect and Mandarin according to claim 1, characterized in that, The convolutional module consists of Swish activation, GLU activation layers, pointwise convolution, depthwise convolution, batch normalization, and normalization layers; the GLU activation function determines which information can be passed to the next module, and its formula is: Where X represents the input, W and V represent different convolution kernels, and b and c are the offsets; The normalization layer integrates information, repositions and optimizes network parameters, and avoids differences in input distribution by dynamically adjusting the length of the input vector. The linear layer connects the neurons in each layer of the neural network to the neurons in the previous layer, realizing linear transformation operations.

3. The method for training a mixed speech recognition model of Lingao dialect and Mandarin according to claim 1, characterized in that, It also includes a multi-head attention decoder, the output of which is connected to the CTC decoder and the multi-head attention decoder. The multi-head attention decoder consists of multiple Transformer-structured decoders, including a multi-head attention layer and a masked multi-head attention layer.

4. The method for training a mixed speech recognition model of Lingao dialect and Mandarin according to claim 3, characterized in that, The multi-head attention decoder decodes in an autoregressive manner. The label output by the decoder in the previous time step is input to the masked multi-head attention layer after passing through the output embedding layer and position encoding, and the corresponding Q, K and V values ​​are generated. Part of the input to the multi-head attention layer is the output K vector and V vector of the encoder, and the other part is the output Q vector of the masked multi-head attention layer. Finally, the complete recognition result is output through the Softmax layer. The position encoding in the multi-head attention decoder uses sine and cosine functions of different frequencies, and its formula is as follows: Where pos refers to the position of the word in the sentence, i represents the dimension of the embedded word vector, 2i represents an even dimension, 2i+1 represents an odd dimension, each dimension of the positional encoding corresponds to a sine signal, and the corresponding wavelength increases geometrically from 2π to 10000▪2π.

5. A training system for a mixed speech recognition model of Lingao dialect and Mandarin, implementing the training method for a mixed speech recognition model of Lingao dialect and Mandarin as described in any one of claims 1 to 4, characterized in that, An end-to-end hybrid speech recognition model is constructed based on the Conformer model. The end-to-end hybrid speech recognition model is an encoder-decoder structure. The encoder is composed of a preset encoder, which contains multiple identical Conformer modules. The decoder adopts a CTC WFST search decoder composed of a CTC decoder and an N-gram language model. The N-gram language model is represented in the form of a weighted finite state converter. The training system for the end-to-end hybrid speech recognition model includes: The input module is used to input a mixed speech signal containing Lingao dialect and Mandarin, and to extract the acoustic feature sequence S of the mixed speech signal; The encoding module is used to input the feature sequence S into a preset encoder, downsample the feature sequence S to reduce feature redundancy information, and then encode it to generate a hidden vector O. The decoding module is used to decode vector O by the CTC WFST search decoder. The CTC Prefix BeamSearch algorithm searches frame by frame in the decoded graph based on the weights on the transition arcs and calculates the weight score in each path. The selection module is used to select the path with the highest score as the optimal path and continuously update the status information. The output module is used to output the label sequence corresponding to the optimal path as the final recognition result Y.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images