Dialect speech recognition and conversion method and device

Abstract

The invention provides a dialect speech recognition and conversion method and device. The method comprises the steps of obtaining dialect speech input data, extracting local features and a global dependency relationship by using a Conformer model, and generating an audio feature sequence. The sequence is input into a shared GRU encoder to generate a hidden state sequence, and the hidden state sequence is transmitted to a CTC decoder of a dialect text and a mandarin text in parallel. And constructing a multi-task learning framework to associate the components, and controlling parameter updating of the components. Through the framework, dialect features are efficiently extracted, and dialect and mandarin texts are generated in parallel. According to the dialect speech recognition and conversion method, the advantages of Conform and CTC-GRU models are combined, and dialect speech recognition and conversion with high accuracy, strong generalization and robustness are realized.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition, and more particularly to a method and device for dialect speech recognition and conversion. Background Art

[0002] Dialects, as crucial symbols of regional culture and ethnic identity, have evolved into a rich and diverse spectrum in China due to a variety of factors, including history, population migration, the natural environment, and climatic conditions. However, significant differences in intonation, pronunciation, speaking speed, and volume among these dialects not only hinder communication between speakers of different dialects but also complicate understanding of the unique characteristics and culture of the region.

[0003] With the acceleration of globalization and the promotion of Mandarin, many dialects are at risk of extinction. Therefore, the research and protection of dialects are particularly important. While existing deep learning-based dialect recognition methods have made some progress in areas such as intelligent voice services, smart elderly care, smart bracelets, dialect identification, and human-computer interaction, they still have many shortcomings.

[0004] For example, current research has not yet fully covered dialects in all regions, especially the recognition capabilities of some county-level dialects and dialects that are on the verge of extinction are limited; the recognition accuracy is low when dealing with dialects with unclear pronunciation and fast speaking speed; at the same time, the existing models have poor generalization and robustness in complex application scenarios, making it difficult to simultaneously recognize dialects across provinces. Summary of the Invention

[0005] The purpose of this application is to overcome the above-mentioned defects in the prior art and provide a dialect speech recognition and conversion method and device.

[0006] This application provides a dialect speech recognition and conversion method, including:

[0007] Get dialect voice input data;

[0008] Based on the dialect speech input data, a Conformer model for extracting local features and global dependencies extracts features and generates an audio feature sequence output by the Conformer;

[0009] Input the audio feature sequence output by the Conformer into a shared GRU encoder to generate a hidden state sequence output by the GRU encoder;

[0010] Passing the hidden state sequence to the CTC decoder for dialect text generation and the CTC decoder for Mandarin text generation in parallel;

[0011] Constructing a multi-task learning framework to associate the shared GRU encoder, the CTC decoder for dialect text generation, and the CTC decoder for Mandarin text generation;

[0012] The multi-task learning framework controls the parameter update process of the shared GRU encoder, the CTC decoder for dialect text generation, and the CTC decoder for Mandarin text generation;

[0013] The CTC decoder generated by the dialect text decodes to generate the dialect text, and the CTC decoder generated by the Mandarin text decodes to generate the Mandarin text, and outputs the dialect text and the Mandarin text.

[0014] Optionally, the Conformer model uses a Mish activation function.

[0015] Optionally, after obtaining the dialect voice input data, the following steps are included:

[0016] An articulation simulation operation is performed on the dialect speech input data, wherein the articulation simulation operation customizes noise injection based on dialect phoneme type characteristics.

[0017] Optionally, the multi-task learning framework includes:

[0018] The parameter update process of the shared GRU encoder is controlled by a weighted focal loss function.

[0019] Optionally, it also includes:

[0020] Pruning and quantization operations are performed on the Conformer model for extracting local features and global dependencies, the shared GRU encoder, the CTC decoder for generating the dialect text, and the CTC decoder for generating the Mandarin text.

[0021] The present application also provides a dialect speech recognition and conversion device, comprising:

[0022] Acquisition module, obtains dialect voice input data;

[0023] An extraction module, based on the dialect speech input data, is used to extract features of a Conformer model for extracting local features and global dependencies, and generate an audio feature sequence output by the Conformer;

[0024] A sequence module that inputs the audio feature sequence output by the Conformer into a shared GRU encoder to generate a hidden state sequence output by the GRU encoder;

[0025] A transfer module transfers the hidden state sequence to a CTC decoder for generating dialect text and a CTC decoder for generating Mandarin text in parallel;

[0026] A framework module constructs a multi-task learning framework to associate the shared GRU encoder, the CTC decoder for dialect text generation, and the CTC decoder for Mandarin text generation;

[0027] A control module, wherein the multi-task learning framework controls the parameter update process of the shared GRU encoder, the CTC decoder for dialect text generation, and the CTC decoder for Mandarin text generation;

[0028] The output module decodes the dialect text using a CTC decoder to generate a dialect text, and decodes the Mandarin text using a CTC decoder to generate a Mandarin text, and outputs the dialect text and the Mandarin text.

[0029] Optionally, the Conformer model uses a Mish activation function.

[0030] Optionally, after the acquisition module acquires the dialect speech input data, the following steps are included:

[0031] An articulation simulation operation is performed on the dialect speech input data, wherein the articulation simulation operation customizes noise injection based on dialect phoneme type characteristics.

[0032] Optionally, the multi-task learning framework includes:

[0033] The parameter update process of the shared GRU encoder is controlled by a weighted focal loss function.

[0034] Optionally, it also includes:

[0035] An optimization module performs pruning and quantization operations on the Conformer model for extracting local features and global dependencies, the shared GRU encoder, the CTC decoder for generating the dialect text, and the CTC decoder for generating the Mandarin text.

[0036] The beneficial effects of this application are:

[0037] The present application provides a method for dialect speech recognition and conversion, comprising: obtaining dialect speech input data; extracting features based on the dialect speech input data using a Conformer model for extracting local features and global dependencies, and generating an audio feature sequence output by the Conformer; inputting the audio feature sequence output by the Conformer into a shared GRU encoder, which encodes and generates a hidden state sequence output by the GRU encoder; transmitting the hidden state sequence in parallel to a CTC decoder for generating dialect text and a CTC decoder for generating Mandarin text; constructing a multi-task learning framework to associate the shared GRU encoder, the CTC decoder for generating dialect text, and the CTC decoder for generating Mandarin text; the multi-task learning framework controls the parameter update process of the shared GRU encoder, the CTC decoder for generating dialect text, and the CTC decoder for generating Mandarin text; the CTC decoder for generating dialect text decodes and generates Mandarin text, and outputs the dialect text and Mandarin text. This application combines the advantages of the Conformer and CTC-GRU models to build a multi-task learning framework, efficiently extract dialect features and generate dialect and Mandarin text in parallel, achieving high accuracy, strong generalization and robust dialect speech recognition and conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of the dialect speech recognition and conversion process in this application;

[0039] Figure 2 It is a schematic diagram of the dialect speech recognition and conversion device in this application. DETAILED DESCRIPTION

[0040] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it is understood that various forms of implementing the present disclosure are not limited by the embodiments set forth herein. Rather, the embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0041] Please refer to Figure 1 As shown, the present application provides a dialect speech recognition and conversion method, comprising:

[0042] S101, obtaining dialect voice input data;

[0043] We collected dialect speech data from various regions of a region through the Chinese Language Resource Collection and Display Platform, manual recordings, and online collection channels. We then simulated unclear pronunciation using professional audio editing software, based on the characteristics of the dialect's phoneme types, such as mispronounced phonemes or syllables.

[0044] The processing reduces or removes the noise in the dialect data, reduces the complexity and computational complexity of subsequent model training, improves training efficiency and facilitates labeling, which is beneficial to data set labeling and model training, verification and testing, thereby reducing feature extraction and training costs and improving dialect recognition accuracy.

[0045] To better meet the joint model's requirements for high data recognition accuracy, strong generalization capabilities, and robustness, after the dialect dataset database is established, the following preprocessing operations are performed on each dialect data in the dialect speech dataset in the database:

[0046] 1. First, perform normalization processing to enhance the generalization ability of this joint model and simplify subsequent processing and model convergence speed, and then perform high-precision noise interference removal of the convolutional neural network (CNN). This also makes it easier for the system to handle different noise environments stably, accurately and quickly.

[0047] 2. Then, operations such as silence removal, windowing, and framing are performed to remove the silent part of the non-speech image and facilitate subsequent feature extraction and processing.

[0048] 3. Finally, based on the endpoint detection method to determine the start and end points of the speech, the Mel-frequency cepstral coefficient (MFCC) is used to extract the characteristics of each dialect speech, providing key information for the subsequent data set annotation.

[0049] The dialect dataset annotation and annotation file format conversion are as follows:

[0050] 3.1. Dialect dataset annotation, in this application, primarily refers to the process of using the dialect speech annotation software Praat to first label the dialect speech dataset preprocessed in the previous step with corresponding relevant labels and semantic labels, timestamp labels, and regional labels to which the corresponding dialect speech belongs, based on the characteristic information such as intonation, pronunciation, speaking speed, and volume of each different region-specific dialect data ultimately extracted. This facilitates the subsequent joint model to accurately identify dialects from different regions and determine the regional affiliation or native place of users of the relevant dialects. For example, the unique pronunciation of the same words and endangered words in different regions is specially noted and accurately recorded, such as their own characteristics and the region in which they are located, so as to accurately identify the pronunciation and regional affiliation of the annotated dialects later. Then, dialect text annotation and Mandarin text annotation are performed on the dialect text dataset and Mandarin text dataset in the database, respectively. Finally, a file containing the annotated dialect data in TextGrid format is saved.

[0051] 3.2. Conversion of the annotation file format: First, ensure that the timestamps in the TextGrid format annotation files corresponding to each region are strictly aligned with the frame time of each dialect data feature extracted using MFCC in the preprocessing; at the same time, the label sequence of the annotation file must match the number of frames of the features extracted using MFCC; and then convert the final optimized annotation file into the format required by the joint model.

[0052] 4. Dialect Dataset Partitioning. Dataset partitioning primarily involves dividing a dataset into two types: training and testing sets, and training, validation, and testing sets. The former is primarily used for performance evaluation, while the latter is used for tuning, enhancing generalization capabilities, and selecting the best-performing model structure. This ultimately creates a more reliable and accurate evaluation system.

[0053] Therefore, to meet the requirements, in the dialect recognition task, the final converted data annotation format files for each dialect in various regions are divided into training set, validation set and test set according to the ratio of 70%, 15% and 15% respectively; and when dividing, in order to promote the generalization ability, recognition accuracy and repeatability of the joint model, it is necessary to ensure that the validation set and the test set remain completely independent.

[0054] S102: extracting features using a Conformer model for extracting local features and global dependencies based on the dialect speech input data, and generating an audio feature sequence output by the Conformer;

[0055] Input the dialect speech input data into the Conformer model: Set the weights using the He initialization method, and the initial value of the bias is 0; Remove noise interference through the convolutional layer and extract local features; Capture global context dependencies through the self-attention mechanism; Output the audio feature sequence of the dialect speech: an integer array representing the ID of the dialect character in the predefined dictionary.

[0056] S103. Input the audio feature sequence output by the Conformer into the shared GRU encoder to generate the hidden state sequence output by the GRU encoder;

[0057] Input the feature sequence output by the Conformer into the shared GRU encoder: Generate the hidden state sequence containing the time-step features and context information; Extract the pronunciation, speech rate, grammar structure, volume, and intonation features of the dialect speech.

[0058] S104. Parallelly transfer the hidden state sequence to the CTC decoder for generating dialect text and the CTC decoder for generating Mandarin text;

[0059] After the shared GRU encoder generates the hidden state sequence containing all the features of the dialect speech, this sequence is simultaneously sent to two independently operating CTC decoders: The CTC decoder for dialect text is specialized in directly converting dialect phonemes into text; The CTC decoder for Mandarin text focuses on mapping the same speech features to standard Mandarin text.

[0060] This parallel architecture realizes the efficient mechanism of "single feature extraction, dual-channel synchronous output". Compared with the traditional serial processing, this design reduces the inference time-consuming, and at the same time ensures that the dialect recognition error will not be transmitted to the Mandarin conversion process - for example, when the dialect decoder misrecognizes the Cantonese character "佢" as "他", the Mandarin decoder can still correctly output "他" according to the original speech features.

[0061] S105. Build a multi-task learning framework to associate the shared GRU encoder, the CTC decoder for generating dialect text, and the CTC decoder for generating Mandarin text;

[0062] The built multi-task learning framework essentially establishes a dynamic cooperation mechanism between components. This framework realizes functional association through three key technologies: Fuse the dialect CTC loss and the Mandarin CTC loss according to a preset ratio to form a unified optimization target; The gradients generated by the dual-channel decoders are synchronously fed back to the shared GRU encoder to drive the encoder to learn the feature expressions that take into account both the dialect characteristics and the Mandarin commonalities; Use the focal loss function to apply higher weights to difficult example samples such as too fast speech rate and unclear pronunciation.

[0063] During the parameter update process, each batch of training data triggers a complete collaborative learning cycle: when the dialect decoder outputs "It's so hot today", the dialect text loss is calculated, and the Mandarin decoder simultaneously generates "It's very hot today" and calculates the Mandarin loss; the two are weightedly fused and back-propagated, allowing the GRU encoder to gradually grasp cross-task rules such as "Cantonese tone features do not affect Mandarin initial consonant recognition."

[0064] S106, the multi-task learning framework controls the parameter update process of the shared GRU encoder, the CTC decoder for generating dialect text, and the CTC decoder for generating Mandarin text;

[0065] Specifically, a multi-task learning framework is constructed to associate the GRU encoder with the dual CTC decoder:

[0066] Dialect text CTC decoder: Receives the GRU hidden state sequence and uses the CTC loss function to decode and generate dialect text.

[0067] Mandarin text CTC decoder: receives the same hidden state sequence and decodes it in the same way to generate Mandarin text.

[0068] The GRU encoder parameters are optimized through the weighted focal loss function, and the loss weight is increased for difficult-to-classify samples.

[0069] S107, the CTC decoder generated by the dialect text decodes to generate the dialect text, the CTC decoder generated by the Mandarin text decodes to generate the Mandarin text, and outputs the dialect text and the Mandarin text;

[0070] The dialect text CTC decoder outputs a dialect text sequence.

[0071] The Mandarin text CTC decoder outputs a Mandarin text sequence.

[0072] Perform post-processing on both texts: remove duplicate characters, correct errors, remove redundant spaces, and add punctuation.

[0073] Based on the dialect character features output by the Conformer model, the GRU classifier is trained, monitored, and evaluated to accurately classify the region to which the dialect belongs.

[0074] To achieve the above purpose of dialect classification, the following operations need to be performed:

[0075] 1. First, add a cross-entropy loss function for regional dialect classification to the GRU model so that the GRU model can learn the corresponding features of each regional dialect as much as possible.

[0076] 2. After calculating the loss between the dialect category predicted by the GRU classifier and the regional label sequence to which the actual dialect speech belongs based on the cross-entropy loss function, the Adam optimizer is first used to update the GRU classifier weights and biases based on the gradient information of the loss function to optimize the cross-loss function, so that the dialect category prediction output by the GRU classifier is closer to the regional label to which the actual dialect speech belongs; and regularly monitor the accuracy, generalization ability, robustness, and F1 value of the classifier on the dialect speech verification set. At the same time, use the warmup learning rate strategy to gradually increase the learning rate to the preset peak value, then keep the learning rate stable and fine-tune parameters such as the number of training rounds in a timely manner based on the performance of the classifier on the verification set.

[0077] When the dialect speech verification set can be used to classify the corresponding dialect category with high accuracy and stability, and the expected generalization ability, robustness, and F1 value are met, training is stopped. Then, the performance of the GRU classifier on the dialect speech test set is evaluated by selecting the same evaluation indicators and using the same calculation method as the verification set.

[0078] 3. If the GRU classifier performs poorly on the above indicators on the validation set and test set, you need to first check the dialect speech validation set and whether there are any imbalanced or difficult-to-classify samples. If so, perform post-processing on the dialect first; then re-execute the operations in the GRU classifier. Otherwise, you need to execute the operations in the GRU classifier first until the GRU classifier meets the above performance standards on the dialect speech validation set.

[0079] 4. If the GRU classifier performs poorly on the above indicators on the dialect speech test set, it is still necessary to first check the dialect speech test set to see if there are any unbalanced or difficult-to-classify samples in the various categories. If so, perform post-processing on the dialect in question first; then re-use the training of the Conformer model and the operations in the GRU classifier. Otherwise, it is necessary to first check the data distribution between the dialect speech test set and the validation set and expand the dialect speech data set, etc., until the GRU classifier achieves the above performance on the dialect speech test set.

[0080] Furthermore, when it is detected that there are class imbalances and difficult-to-classify samples in the dialect speech dataset, the accuracy of classifying dialect speech by the joint model of the shared Conformer combined with the GRU classifier is improved.

[0081] For the problem of class imbalance, the following actions are taken:

[0082] 1. First use the weighted cross entropy loss function to replace the original cross entropy loss function in the GRU classifier.

[0083] 2. For categories that are significantly smaller in number than other dialect categories, initially set a larger weight.

[0084] 3. Use the reset GRU classifier loss function and weights to start training the GRU classifier and continue to evaluate the performance of the GRU classifier in terms of accuracy, generalization ability, robustness, and F1 value on the dialect speech verification set and the dialect speech test set. If the effect on the dialect speech verification set and the test set is still poor, continue to adjust the weights until the GRU achieves stable and high accuracy, generalization ability, robustness, and F1 value on the dialect speech verification set and the test set.

[0085] For samples that are difficult to classify, the following actions are taken:

[0086] 1. Add the focal loss function to the GRU classifier that already has a weighted cross entropy loss function.

[0087] 2. First, continuously adjust the convolution kernel size of the Conformer network and monitor and evaluate the Conformer's performance, such as accuracy, generalization ability, robustness, and F1 value, on the dialect speech validation set and test set until the most appropriate convolution kernel size is selected and maintained for the model; this allows the Conformer to better extract local features of the two difficult-to-classify Tianjin Tanggu District and Wuqing District dialect speech samples.

[0088] 3. Using the focal loss function, we continuously manually reduce the loss weights of other easily classified dialect speech datasets; and increase the loss contribution to dialect and dialect speech samples that are difficult to classify. We continue to monitor and evaluate the performance of the GRU classifier in terms of accuracy, generalization ability, robustness, and F1 value on the dialect speech validation set and test set; until the best two focal loss function parameters are selected, so as to better extract the local features of the two categories of samples and make the GRU classifier pay more attention to the two difficult-to-classify dialect speech samples.

[0089] Model deployment optimization:

[0090] For Conformer, GRU encoder and dual CTC decoder:

[0091] Pruning: removing unimportant weights or connections from a model.

[0092] Quantization operation: Convert floating-point weights and activation values ​​to low-precision integer form.

[0093] Furthermore, the Conformer with the best performance described above is integrated with a joint model based on two CTC decoders sharing a GRU encoder and a Conformer-GRU classifier into a practical application environment (such as a server or embedded device) for real-time or batch processing. To meet the low memory and application requirements of the target device, the following operations need to be performed on the deployed model before deployment:

[0094] 1. Pruning mainly refers to removing unimportant weights or connections in the model to reduce model complexity.

[0095] 2. Quantization mainly refers to converting the floating-point weights and activation values ​​in the model into low-precision integer form.

[0096] 3. Model conversion. This mainly refers to the need to use appropriate tools to convert the best-performing model into a format executable in the actual application environment if the target deployment environment is not suitable for the framework used during training. For the PyTorch framework used this time, ONNX tools are first used to convert the evaluated joint model mentioned above into an ONNX format suitable for execution under multiple deep learning frameworks or hardware platforms.

[0097] 4. Select an embedded device or server and install the cross-compilation tool chain.

[0098] Choosing an embedded device or server mainly refers to ensuring that it has sufficient computing resources and storage space to support the operation of the model based on the above-mentioned evaluated model;

[0099] Installing a cross-compilation toolchain is mainly to compile the model and deep learning framework into binary files that can be run in the actual application environment. If the target device is a server or a high-performance environment, cross-compilation is not required; instead, you need to install the deep learning framework PyTorch and its corresponding dependencies.

[0100] At this point, you can decide how to test and run the converted model based on the specific application scenario and requirements. For example:

[0101] In scenarios where real-time or batch data processing is required, dialect recognition and classification tasks are performed by sequentially loading the Conformer, a joint model of two CTC decoders sharing a GRU encoder, and a GRU classifier, and calling the corresponding inference interface to perform inference services;

[0102] In a restricted application environment, the converted Conformer needs to be run directly in sequence. The two CTC decoders share a joint model of a GRU encoder and a GRU classifier to perform the tasks of dialect recognition and classification.

[0103] like Figure 2 As shown, the present application also provides a dialect speech recognition and conversion device, comprising:

[0104] Acquisition module 201, acquiring dialect speech input data;

[0105] Extraction module 202, based on the dialect speech input data, extracts features using a Conformer model for extracting local features and global dependencies, and generates an audio feature sequence output by the Conformer;

[0106] The sequence module 203 inputs the audio feature sequence output by the Conformer into a shared GRU encoder to generate a hidden state sequence output by the GRU encoder;

[0107] A transfer module 204 transfers the hidden state sequence to a CTC decoder for generating dialect text and a CTC decoder for generating Mandarin text in parallel;

[0108] Framework module 205, constructing a multi-task learning framework to associate the shared GRU encoder, the CTC decoder for dialect text generation, and the CTC decoder for Mandarin text generation;

[0109] Control module 206, the multi-task learning framework controls the parameter update process of the shared GRU encoder, the CTC decoder for dialect text generation, and the CTC decoder for Mandarin text generation;

[0110] The output module 207 decodes the dialect text using the CTC decoder to generate the dialect text, and decodes the Mandarin text using the CTC decoder to generate the Mandarin text, and outputs the dialect text and the Mandarin text.

[0111] Optionally, the Conformer model uses a Mish activation function.

[0112] Optionally, after the acquisition module acquires the dialect speech input data, the following steps are included:

[0113] An articulation simulation operation is performed on the dialect speech input data, wherein the articulation simulation operation customizes noise injection based on dialect phoneme type characteristics.

[0114] Optionally, the multi-task learning framework includes:

[0115] The parameter update process of the shared GRU encoder is controlled by a weighted focal loss function.

[0116] Optionally, it also includes:

[0117] An optimization module performs pruning and quantization operations on the Conformer model for extracting local features and global dependencies, the shared GRU encoder, the CTC decoder for generating the dialect text, and the CTC decoder for generating the Mandarin text.

[0118] The above description of the embodiments is intended to facilitate understanding and application of this application by those skilled in the art. It will be readily apparent to those skilled in the art that various modifications to the above embodiments can be made, and the general principles described herein can be applied to other embodiments without requiring creative effort. Therefore, this application is not limited to the above embodiments. Any improvements or modifications made to this application by those skilled in the art based on the disclosure of this application should fall within the scope of protection of this application.

Claims

1. A dialect speech recognition and conversion method, characterized in that: include: Get dialect voice input data; Based on the dialect speech input data, extract features through a Conformer model that extracts local features and global dependencies, and generate an audio feature sequence output by the Conformer; Input the audio feature sequence output by the Conformer into a shared GRU encoder to generate a hidden state sequence output by the GRU encoder; Passing the hidden state sequence to the CTC decoder for dialect text generation and the CTC decoder for Mandarin text generation in parallel; Constructing a multi-task learning framework to associate the shared GRU encoder, the CTC decoder for dialect text generation, and the CTC decoder for Mandarin text generation; The multi-task learning framework controls the parameter update process of the shared GRU encoder, the CTC decoder for dialect text generation, and the CTC decoder for Mandarin text generation; The CTC decoder generated by the dialect text decodes to generate the dialect text, and the CTC decoder generated by the Mandarin text decodes to generate the Mandarin text, and outputs the dialect text and the Mandarin text.

2. The method for dialect speech recognition and conversion according to claim 1, wherein: The Conformer model uses the Mish activation function.

3. The method for dialect speech recognition and conversion according to claim 1, wherein: After obtaining dialect voice input data, including: An articulation simulation operation is performed on the dialect speech input data, wherein the articulation simulation operation customizes noise injection based on dialect phoneme type characteristics.

4. The method for dialect speech recognition and conversion according to claim 1, wherein: The multi-task learning framework includes: The parameter update process of the shared GRU encoder is controlled by a weighted focal loss function.

5. The method for dialect speech recognition and conversion according to claim 1, wherein: Also includes: Pruning and quantization operations are performed on the Conformer model for extracting local features and global dependencies, the shared GRU encoder, the CTC decoder for generating the dialect text, and the CTC decoder for generating the Mandarin text.

6. A dialect speech recognition and conversion device, characterized in that: include: Acquisition module, obtains dialect voice input data; An extraction module, based on the dialect speech input data, is used to extract features of a Conformer model for extracting local features and global dependencies, and generate an audio feature sequence output by the Conformer; A sequence module that inputs the audio feature sequence output by the Conformer into a shared GRU encoder to generate a hidden state sequence output by the GRU encoder; A transfer module transfers the hidden state sequence to a CTC decoder for generating dialect text and a CTC decoder for generating Mandarin text in parallel; A framework module constructs a multi-task learning framework to associate the shared GRU encoder, the CTC decoder for dialect text generation, and the CTC decoder for Mandarin text generation; A control module, wherein the multi-task learning framework controls the parameter update process of the shared GRU encoder, the CTC decoder for dialect text generation, and the CTC decoder for Mandarin text generation; The output module decodes the dialect text using a CTC decoder to generate a dialect text, and decodes the Mandarin text using a CTC decoder to generate a Mandarin text, and outputs the dialect text and the Mandarin text.

7. The dialect speech recognition and conversion device according to claim 6, characterized in that: The Conformer model uses the Mish activation function.

8. The dialect speech recognition and conversion device according to claim 6, characterized in that: After the acquisition module acquires the dialect voice input data, it includes: An articulation simulation operation is performed on the dialect speech input data, wherein the articulation simulation operation customizes noise injection based on dialect phoneme type characteristics.

9. The dialect speech recognition and conversion device according to claim 6, characterized in that: The multi-task learning framework includes: The parameter update process of the shared GRU encoder is controlled by a weighted focal loss function.

10. The dialect speech recognition and conversion device according to claim 6, characterized in that: Also includes: An optimization module performs pruning and quantization operations on the Conformer model for extracting local features and global dependencies, the shared GRU encoder, the CTC decoder for generating the dialect text, and the CTC decoder for generating the Mandarin text.

Images