Dialect speech recognition method, system and model

Abstract

The present application belongs to the technical field of speech recognition, and particularly relates to a dialect speech recognition method, system and model. The dialect speech recognition method comprises: S1, generating pseudo labels and screening by a plurality of teacher models; S2, sub-model training; S3, data fusion; S4, student model training, repeating steps S3 to S4 until round fusion training is performed, and student models of each region are obtained after training. The present application combines multi-teacher model knowledge distillation and K nearest neighbor dynamic data fusion training to construct an end-to-end speech transcription method suitable for a multi-dialect scene, and is particularly suitable for a less resourceful dialect environment with a complex tone system, rich speech variants and scarce data.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of speech recognition, and particularly relates to a dialect speech recognition method, system and model. BACKGROUND

[0002] As an important branch of speech recognition technology, dialect speech recognition has received more and more attention in recent years. The main task of dialect speech transcription is to automatically recognize and transcribe the input dialect speech signal, and output the standard written text corresponding to the speech, so as to realize the structured and standardized storage and further utilization of dialect corpus.

[0003] In dialect speech transcription, the following task objectives and technical challenges are mainly faced:

[0004] Firstly, in order to accurately transcribe dialect speech signals, the phonemes, tones, and prosodic features in the dialect speech signals need to be fully modeled and distinguished, especially in the case of homophonic words with different tones, to ensure the semantic correctness of the transcribed text. For example, Guizhou Province is located in the southwest of China, with complex terrain and diverse dialect regions, mainly divided into three major sub-dialect regions: central Guizhou, northern Guizhou, and southwestern Guizhou. Each region contains a large number of subdivided variants, and there are more than 15 tone variants in southeastern Guizhou alone. The slight difference in tone can cause changes in word meaning. This high degree of diversity poses a great challenge to automatic speech recognition (ASR) systems, so the model needs to have strong tone discrimination ability.

[0005] Secondly, there may be a situation where multiple dialect variants are mixed within a certain regional range, and a single model often cannot cover the speech features of all dialect regions. For example, in cross-domain testing, the recognition performance of the model will decrease significantly, resulting in a significant increase in character error rate (CER). Therefore, the task of dialect transcription not only requires the model to have good recognition ability within a single dialect region, but also requires the model to have strong cross-domain generalization ability.

[0006] In addition, existing dialect speech transcription methods rely on large-scale high-quality labeled data, but dialect corpus has the problems of low coverage and high labeling cost, which makes it difficult for current models based on mainstream architectures such as Conformer-CTC to fully learn the fine-grained difference features of dialect speech in the training process. When facing new regions and new speakers, the model is prone to recognition bias and unstable results.

[0007] Therefore, in the task of dialect speech transcription, the following technical problems need to be solved:

[0008] (1) Under low-resource conditions, effectively model the complex tone and phonological structure of dialect speech, and improve the model's ability to distinguish fine-grained tone variants;

[0009] (2) Enhance the cross-domain robustness and generalization performance of the model in the presence of significant acoustic and language regional drift;

[0010] (3) Reduce the model's dependence on large-scale labeled corpus under limited labeled data conditions, and realize unified modeling and recognition of multiple dialectal variants. SUMMARY

[0011] The present application aims to provide a dialectal speech recognition method, system and model to solve the technical problems of low accuracy, poor generalization ability and high labeling cost in existing dialectal speech recognition technology.

[0012] A dialectal speech recognition method, the dialectal speech recognition method comprising:

[0013] S1, obtaining regional data sets of several regions, the regional data sets comprising several unlabeled dialectal speech samples, using several teacher models of different structures to infer all the dialectal speech samples, obtaining multiple candidate labels corresponding to the dialectal speech samples, selecting pseudo labels corresponding to the dialectal speech samples from the multiple candidate labels, and obtaining several regional data sets comprising several dialectal speech samples with pseudo labels;

[0014] S2, initializing and training a sub-model of each region using the regional data set of each region, and obtaining a trained sub-model of each region;

[0015] S3, calculating the recognition accuracy of the sub-model of each region on the data set of other regions to obtain a dialectal similarity matrix, selecting one of the other regions with the highest recognition accuracy as a neighbor region in the dialectal similarity matrix, and integrating the regional data set corresponding to the neighbor region into the regional data set of the current region to form an enhanced regional data set;

[0016] S4, training a student model of each region using the enhanced regional data set of each region, repeating steps S3 to S4 until rounds of fusion training are performed to obtain a trained student model of each region.

[0017] Optionally, in step S1, after obtaining the regional data sets of several regions, before using several teacher models of different structures to infer all the dialectal speech samples, it further comprises:

[0018] preprocessing all the dialectal speech samples, the preprocessing comprising one or more combinations of speech length filtering, voice activity detection (VAD) segmentation, audio denoising, volume normalization and feature extraction;

[0019] Preferably, dialect speech samples less than 0.5 seconds and greater than 20 seconds are removed when performing speech length filtering;

[0020] Preferably, the target volume is -20 dB when performing volume normalization processing.

[0021] Preferably, fbank feature extraction based on an 80-dimensional mel spectrum filter bank is used when performing feature extraction.

[0022] Optionally, in step S1, three different structures of teacher models are used to infer all the dialect speech samples, and the three teacher models are Paraformer model, DeepSpeech2 model and SenseVoiceSmall model respectively.

[0023] Optionally, in step S1, the pseudo label corresponding to the dialect speech sample is selected from a plurality of candidate labels, comprising:

[0024] The plurality of candidate labels corresponding to each dialect speech sample are input into a pre-set large language model (LLM), and in the large language model, the pseudo label corresponding to each dialect speech sample is selected by a pre-set prompt word.

[0025] Optionally, the large language model is Qwen2.5-72B or a general-purpose language understanding and generation model with equivalent performance.

[0026] Optionally, the prompt word contains one or a combination of more than one of dialect context consistency judgment, semantic rationality detection, syllable matching degree analysis and regional geographic location information.

[0027] Optionally, in step S1, the several regions are different counties or cities within the same province, different administrative districts within the same municipality or different cities within the same autonomous region.

[0028] Optionally, in step S2, the sub-models are initialized and trained using the regional data sets of each region and part of the artificial annotation data sets of each region.

[0029] Optionally, in step S2, the sub-models are Conformer-CTC structure sub-models.

[0030] Optionally, the sub-models include an encoder and a decoder, the encoder includes a multi-head self-attention module, a convolution module, a feedforward network module, a layer normalization and residual connection module connected in sequence, and the decoder uses a connection time classification (CTC) decoding layer.

[0031] The model input of the sub-model is a dialect speech sample or a preprocessed dialect speech sample, and the model output of the sub-model is a transcription text corresponding to the dialect speech sample.

[0032] Optionally, in step S2, the training target when the sub-model is initialized and trained is to minimize a CTC loss function.

[0033] Optionally, in step S2, when the sub-model is initialized and trained, one or more combination strategies of pre-training model loading, optimizer configuration and parallel training are adopted.

[0034] The pre-training model loading strategy is to load pre-training weights on a Mandarin speech recognition dataset when initializing the sub-model; the Mandarin speech recognition dataset is preferably at least one of AISHELL-1 and WeTSpeech;

[0035] The optimizer configuration strategy is to use an Adam optimizer combined with SpecAugment for data enhancement; preferably, the initial learning rate of the Adam optimizer is 1e-4, and the warmup ratio is 0.1.

[0036] The parallel training strategy is that the sub-models corresponding to each region are respectively trained in parallel on a multi-card GPU cluster.

[0037] Optionally, in step S3, the region data set of region i is used as a test set, and the region data set of region j is used as a training set. The sub-model of region j is used to recognize the region data set of region i , to obtain a transcription text , and the transcription text is compared with the transcription text obtained by recognizing the region data set of region i on the sub-model corresponding to itself , to obtain the recognition accuracy of the sub-model on the region data set , wherein i≠j.

[0038] The calculation formula of the recognition accuracy is:

[0039]

[0040]

[0041]

[0042] wherein, WER represents the writing text and the transcription text​ the word error rate between the transcription text and the written text, S1 represents the number of words of substitution errors, D1 represents the number of words of deletion errors, I1 represents the number of words of insertion errors, and N1 represents the total number of words in the reference text; CER represents the writing text and the transcription text , S2 represents the number of characters of substitution errors, D2 represents the number of characters of deletion errors, I2 represents the number of characters of insertion errors, and T represents the total number of characters in the reference text;

[0043] all cross-validation results are constituted into a dialect similarity matrix of dimension N . , then:

[0044]

[0045] For region i, in the dialect similarity matrix , the region j with the highest recognition accuracy except for itself is selected as a neighbor region, and the region data set corresponding to the neighbor region is merged into the region data set to form an enhanced region data set .

[0046] Optionally, in the dialect similarity matrix , elements lower than a preset similarity limit threshold are set to zero.

[0047] Optionally, after selecting, in step S3, one other region with the highest recognition accuracy as a neighbor region, and before merging, in the region data set corresponding to the neighbor region, into the region data set of the current region, the method further comprises:

[0048] determining whether the highest recognition accuracy selected is not less than a preset dynamic threshold, and when it is not less than, merging, in the region data set corresponding to the neighbor region, into the region data set of the current region, otherwise, giving up the merging.

[0049] Optionally, the calculation formula of the dynamic threshold is as follows:

[0050]

[0051]

[0052]

[0053]

[0054] wherein, represents the mean of the recognition accuracies of all neighbor regions of region i,​​ denotes the standard deviation of the recognition accuracy of all neighboring regions of region i, denotes an adjustable parameter, denotes a preset minimum threshold value, denotes the i-th round, denotes the set of all neighboring regions filtered out in the dialect similarity matrix denotes the set of all neighboring regions filtered out in the dialect similarity matrix denotes the set of all neighboring regions filtered out in the dialect similarity matrix denotes the set of all neighboring regions filtered out in the dialect similarity matrix.

[0055] Optionally, the calculation formula of the dynamic threshold is as follows:

[0056]

[0057]

[0058]

[0059]

[0060] wherein, denotes the mean of the recognition accuracy of all neighboring regions of region i, denotes the standard deviation of the recognition accuracy of all neighboring regions of region i, denotes a preset minimum threshold value, denotes the i-th round, denotes the set of all neighboring regions filtered out in the dialect similarity matrix denotes the set of all neighboring regions filtered out in the dialect similarity matrix denotes the set of all neighboring regions filtered out in the dialect similarity matrix denotes the set of all neighboring regions filtered out in the dialect similarity matrix.

[0061] Optionally, in step S3, sample balancing processing is performed on the enhanced region dataset, and the sample balancing processing is performed in the following manner: if the number of dialect speech samples in any region dataset is greater than a preset number, a part of the dialect speech samples are randomly deleted until the number of dialect speech samples in the region dataset is not greater than the preset number.

[0062] Optionally, the student model adopts a model with the same Conformer-CTC structure as the sub-model.

[0063] Optionally, the loss function of the student model adopts a multi-objective joint distillation loss function, and the multi-objective joint distillation loss function is defined as:

[0064]

[0065] wherein, representing an enhanced regional dataset a KL divergence loss between the student model output distribution U and the pre-set teacher model prediction distribution, is a CTC loss, is a cross-entropy loss, is a weight hyperparameter.

[0066] Optionally, the pre-set teacher model is a random one or the one with the best performance among the teacher models of different structures in step S1, where the best performance refers to the teacher model that contributes the most pseudo labels in step S1;

[0067] Optionally, the pre-set teacher model is a SenseVoice Small model.

[0068] A dialect speech recognition system, comprising:

[0069] a data processing module configured to obtain regional datasets of a plurality of regions, the regional datasets comprising a plurality of unlabeled dialect speech samples, and perform preprocessing on all the dialect speech samples, the preprocessing comprising a combination of one or more of speech length filtering, speech activity detection segmentation, audio denoising, volume normalization, and feature extraction;

[0070] a multi-teacher model pseudo label generation module configured to perform inference on all the dialect speech samples using a plurality of teacher models of different structures to obtain a plurality of candidate labels corresponding to the dialect speech samples;

[0071] a large language model screening module configured to screen a pseudo label corresponding to the dialect speech sample from the plurality of candidate labels to obtain a plurality of regional datasets comprising a plurality of dialect speech samples with pseudo labels;

[0072] a sub-model initialization module configured to initialize and train a respective sub-model using the regional dataset of each region to obtain a trained sub-model of each region;

[0073] a dialect similarity calculation module configured to calculate the recognition accuracy of the sub-model of each region on the regional dataset of another region to obtain a dialect similarity matrix;

[0074] a K-nearest neighbor dynamic data fusion module configured to select, in the dialect similarity matrix, one other region with the highest recognition accuracy for the current region as a neighbor region, and combine the regional dataset corresponding to the neighbor region into the regional dataset of the current region to form an enhanced regional dataset;

[0075] A joint distillation training module is used to train respective student models using the enhanced regional data sets of respective regions, and multi-round dynamic training is performed using the dialect similarity calculation module and the K nearest neighbor dynamic data fusion module until round fusion training is performed to obtain the respective student models trained for respective regions.

[0076] A dialect speech recognition model adopts the student model obtained by the dialect speech recognition method.

[0077] Beneficial effects: The present application has at least one or more of the following advantages:

[0078] 1. The present application combines multi-teacher model knowledge distillation and K nearest neighbor (KNN) dynamic data fusion training to construct an end-to-end speech transcription method suitable for a multi-dialect scene. The method is particularly suitable for a less resourceful dialect environment with a complex tone system, rich speech variants, and scarce data.

[0079] 2. To overcome the problem of scarce dialect annotation data, the present application proposes a multi-teacher model pseudo-label distillation scheme. The scheme combines the structural diversity of multiple teacher models and the powerful semantic understanding ability of large language models to significantly improve the quality of pseudo-labels, reduce the cost of manual annotation, and ensure that the screening results are not only accurate but also have local expression characteristics.

[0080] 3. After obtaining a large-scale high-confidence pseudo-label data set, the present application further proposes a dynamic data fusion method based on the K nearest neighbor (KNN) algorithm to mine the similarity between dialects of each county and city, form an optimal regional combination, and significantly improve the generalization ability of the model.

[0081] 4. The present application uses a multi-objective joint distillation loss function. The student model not only learns the soft label information of the teacher model but also considers the end-to-end sequence prediction ability and label consistency to improve the model performance.

[0082] 5. The present application designs a multi-round progressive training process. After each round of training, the dialect similarity matrix and the enhanced regional data set are dynamically updated. The "data distillation + KNN dynamic data fusion" strategy is combined to gradually optimize the training data distribution and model ability of each regional sub-model, significantly enhancing the modeling and generalization ability of the model for regional dialect acoustics and language characteristics. Combined with multi-objective joint distillation training, the challenges brought by dialect data scarcity and diversity are effectively solved, and the performance and practical value of the dialect speech recognition method, system, and model are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0083] Figure 1 A flowchart of the present application.

[0084] Figure 2 A flow chart for the whole process of embodiment 1 of the present application;

[0085] Figure 3 A flow chart for steps S2 and S3 in embodiment 1 of the present application. DETAILED DESCRIPTION

[0086] The preferred embodiments of the present application will be described in detail with reference to the drawings, so as to make the objects, features and advantages of the present application more clearly understood. It should be understood that the embodiments shown in the drawings are not intended to limit the scope of the present application, but merely to illustrate the essential spirit of the technical solutions of the present application.

[0087] In the following description, for the purpose of explaining various disclosed embodiments, certain specific details are set forth in order to provide a thorough understanding of various disclosed embodiments. However, one skilled in the relevant arts will recognize that embodiments can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, materials, or operations are not shown or described in detail in order to avoid obscuring the description of embodiments.

[0088] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0089] In the following description, for the purpose of clarity, directional terms such as "front", "back", "left", "right", "upper", "lower", "over", "under", "above", "below", etc. are used with reference to the orientation of the drawings. However, it is to be understood that the application can assume various alternative orientations and, accordingly, such terms are not to be understood as limiting.

[0090] With reference to Figure 1 The embodiment of the present application provides a dialect speech recognition method, which comprises the following steps:

[0091] S1, pseudo label generation and screening: obtaining regional data sets of a plurality of regions, the regional data sets comprising a plurality of unlabeled dialect speech samples, using a plurality of teacher models with different structures to infer all dialect speech samples to obtain a plurality of candidate labels corresponding to the dialect speech samples, screening pseudo labels corresponding to the dialect speech samples from the plurality of candidate labels, and obtaining a plurality of regional data sets comprising a plurality of dialect speech samples with pseudo labels.

[0092] Each region has its own independent regional data set, and each regional data set contains multiple unlabeled dialect speech samples. In this step, based on the knowledge distillation method of the multi-teacher model, multiple pseudo-labels are generated and screened through the multi-teacher model, so that a single dialect speech sample realizes pseudo-labeling from no label, reduces the cost of manual labeling, and overcomes the problem of lack of dialect labeled data.

[0093] In an embodiment, in step S1, after obtaining the regional data sets of several regions, before using several teacher models with different structures to infer all dialect speech samples, the all dialect speech samples are preprocessed to ensure the data quality of the input model is stable and reliable.

[0094] The preprocessing includes one or more combinations of speech length filtering, voice activity detection (VAD) segmentation, audio denoising, volume normalization, and feature extraction.

[0095] When performing speech length filtering, it is preferred to remove dialect speech samples less than 0.5 seconds and greater than 20 seconds; when performing volume normalization, the target volume is preferably -20dB; when performing feature extraction, it is preferred to use fbank feature extraction based on an 80-dimensional mel-spectral filter bank.

[0096] In an embodiment, in step S1, three teacher models with different structures are used to infer all dialect speech samples, and the three teacher models are Paraformer model, DeepSpeech2 model and SenseVoice Small model.

[0097] The Paraformer model is proposed by the Alibaba DAMO Academy, which is a high-efficiency real-time speech recognition model based on a non-autoregressive Transformer structure. It significantly improves decoding efficiency through parallel decoding and adapts to real-time speech recognition scenarios.

[0098] The DeepSpeech2 model is an end-to-end CTC (Connectionist Temporal Classification) structured speech recognition model proposed by Baidu, which has strong acoustic modeling capabilities and adapts to multi-dialect features.

[0099] The SenseVoice Small model is a SenseVoice open source model, which is a multi-task multi-language audio understanding model with speech recognition, language recognition, speech emotion recognition, and acoustic event detection capabilities.

[0100] By using multiple teacher models with different structures and pre-training corpora, the coverage and diversity of the speech content can be significantly improved, providing rich candidates for subsequent screening.

[0101] In an embodiment, in step S1, the pseudo label corresponding to the dialect speech sample is screened from the plurality of candidate labels, comprising: inputting the plurality of candidate labels corresponding to each dialect speech sample into a preset large language model (LLM), and screening in the large language model by using a preset prompt word, so that the large language model screens the pseudo label corresponding to each dialect speech sample.

[0102] In the present embodiment, by inputting the plurality of candidate labels of the same dialect speech sample into the large language model as a word dialogue, the large language model is guided by the prompt word to output the optimal label as the pseudo label. The final result output by the large language model is the high-confidence dialect speech annotation after screening.

[0103] In an embodiment, the large language model is Qwen2.5-72B or a general-purpose language understanding generation model with equivalent performance.

[0104] In an embodiment, the prompt word comprises one or more combinations of dialect context consistency judgment, semantic rationality detection, syllable matching degree analysis, and regional geographic location information.

[0105] The prompt word is a prompt word designed in advance by a person according to local dialect characteristics for screening a plurality of candidate labels. The large language model can be guided by a script to output the optimal label as the pseudo label. Since the regional data set is classified by region, the regional geographic location information of the plurality of dialect speech samples in the regional data set can be annotated when the regional data set is obtained, so that the prompt word can obtain the regional geographic location information corresponding to each dialect speech sample by a script and prompt the large language model.

[0106] In specific implementation, the dialect context consistency judgment requires the model to judge whether the output is a commonly used dialect vocabulary in the target region according to the context; the semantic rationality detection requires the output to be complete and avoid partial missing words in the teacher model; and the syllable matching degree analysis requires the output syllable to have the highest correspondence rate with the audio transcription result.

[0107] In an embodiment, in step S1, the several regions are different counties or cities in the same province, different administrative regions in the same municipality, or different cities in the same autonomous region.

[0108] S2, sub-model training: using the regional data set of each region to initialize and train the respective sub-model, to obtain the respective trained sub-model of each region.

[0109] After obtaining the regional dataset with high-confidence pseudo-labels, each region initializes and trains its own sub-model based on its respective regional dataset. In this step, there are as many sub-models as there are regions, and each sub-model is initialized and trained independently.

[0110] In one embodiment, in step S2, the regional datasets of each region and the partially manually labeled datasets of each region are used to initialize and train their respective sub-models.

[0111] In this embodiment, in addition to the regional dataset with high-confidence pseudo-labels obtained in step S1, there may also be some manually labeled datasets with collected annotation data. These manually labeled datasets will also be used for initial training.

[0112] In one embodiment, in step S2, the sub-model adopts a Conformer-CTC structure.

[0113] The Conformer-CTC architecture combines the advantages of the local feature modeling capability of convolutional neural networks (CNNs) with the global dependency modeling capability of the Transformer architecture, and can fully establish the local and global dependencies of speech.

[0114] In one embodiment, the sub-model includes an encoder and a decoder. The encoder includes a multi-head self-attention module, a convolution module, a feedforward network module, and a layer normalization and residual connection module connected in sequence. The decoder adopts a connection-temporal classification (CTC) decoding layer. The input of the sub-model is a dialect speech sample or a preprocessed dialect speech sample, and the output of the sub-model is the transcribed text corresponding to the dialect speech sample.

[0115] Multi-head self-attention module: used to capture long-distance dependencies in the input speech sequence, enhancing the ability to understand the context of the whole sentence.

[0116] Convolution Module: Used to extract key local information and enhance the ability to express short-term features such as phonemes and words.

[0117] Feedforward Module: Used to enhance the nonlinear expressive power of the model and improve its ability to fit complex distributions.

[0118] Layer Normalization and Residual Connection Module: This module adds normalization and residual paths between modules to stabilize the training process and accelerate model convergence.

[0119] The output layer adopts a CTC decoding layer, which is used to calculate a CTC loss function between a predicted sequence and a real label sequence, realize alignment-free training, and improve the adaptability of the model in a dialect speech transcription task.

[0120] In an embodiment, in step S2, the training target when the sub-model is initialized is to minimize the CTC loss function.

[0121] The specific formula of the CTC loss function is as follows:

[0122]

[0123] wherein, is an input dialect speech sample, is a corresponding sample label sequence, and N is the number of samples.

[0124] In an embodiment, in step S2, when the sub-model is initialized, one or more combination strategies of pre-training model loading, optimizer configuration, and parallel training are used to enhance the performance and stability of the model.

[0125] The pre-training model loading strategy is to load pre-training weights on a Mandarin speech recognition dataset when initializing the sub-model, thereby shortening the convergence time. The Mandarin speech recognition dataset is preferably at least one large-scale Mandarin speech recognition dataset in AISHELL-1 and WeTSpeech.

[0126] The optimizer configuration strategy is to use an Adam optimizer combined with SpecAugment for data enhancement. Preferably, the initial learning rate of the Adam optimizer is 1e-4, and the warmup ratio is 0.1.

[0127] The parallel training strategy is to train each sub-model corresponding to each region in parallel on a multi-card GPU cluster, thereby significantly shortening the overall initialization time and improving the training efficiency.

[0128] S3, data fusion: calculate the recognition accuracy of each sub-model of each region on the data set of other regions to obtain a dialect similarity matrix, select one of the other regions with the highest recognition accuracy in the dialect similarity matrix as a neighbor region, and integrate the data set of the neighbor region into the data set of the current region to form an enhanced regional data set.

[0129] After the initialization training of the sub-models of each region in step S2 is completed, this step enters the KNN dynamic data fusion phase, which aims to mine the similarity between dialects of each region to realize dynamic data fusion and improve the generalization ability of the model.

[0130] In one embodiment, in step S3, cross-validation is used to process the regional dataset of region i. Sub-model in regions j other than region i ( The text is transcribed by recognizing the text on the screen. The region dataset of region i In its corresponding sub-model The written text obtained by recognition With transcribed text By comparing, sub-models are obtained. Regional datasets Recognition accuracy ,in,

[0131] The formula for calculating recognition accuracy is:

[0132]

[0133]

[0134]

[0135] WER stands for Written Text With transcribed text The word error rate (WER) is the ratio of words between different words. A lower WER value indicates better model performance. S1 represents the number of incorrectly replaced words, D1 represents the number of incorrectly deleted words, I1 represents the number of incorrectly inserted words, and N1 represents the total number of words in the reference text. CER represents the written text. With transcribed text The character error rate (CER) is the ratio of the number of characters replaced by the model. The lower the CER value, the better the model's performance. S2 represents the number of characters replaced incorrectly, D2 represents the number of characters deleted incorrectly, I2 represents the number of characters inserted incorrectly, and T represents the total number of characters in the reference text.

[0136] All cross-validation results constitute Dialect similarity matrix ,but:

[0137]

[0138] Here, setting the diagonal elements of the matrix to 0 indicates that the results of the self-model on its own dataset are not considered.

[0139] For region i, in the dialect similarity matrix In the dataset, the region j with the highest recognition accuracy (excluding itself) is selected as the neighboring region, and the corresponding region datasets are then processed. Merging into regional datasets merge the region data set of the neighbor region into the region data set of the current region to form an enhanced region data set .

[0140] In an embodiment, in the dialect similarity matrix , elements lower than a preset similarity limit threshold are set to zero.

[0141] In this embodiment, by setting elements lower than a preset similarity limit threshold to zero, low-quality model results are avoided from interfering with the KNN screening process.

[0142] The preset similarity limit threshold is preferably 50%, but can also be changed according to the actual scene.

[0143] In an embodiment, after selecting, in step S3, one other region with the highest recognition accuracy of the current region as a neighbor region, before merging the region data set corresponding to the neighbor region into the region data set of the current region to form an enhanced region data set, the method further includes: determining whether the highest recognition accuracy is not less than a preset dynamic threshold, and when it is not less than, merging the region data set corresponding to the neighbor region into the region data set of the current region, otherwise, abandoning the merging.

[0144] After the dialect similarity matrix is constructed, the embodiment screens the other region data set most similar to the dialect of the current region through the KNN algorithm, and generates an enhanced data set based on a dynamic threshold fusion, which is used for subsequent joint training.

[0145] The dynamic threshold of region i is set as , and only when the neighbor region satisfies the similarity condition , the region data set is merged with the region data set of the current region to form an enhanced data set:

[0146]

[0147]

[0148] wherein, denotes the set of all neighbor regions screened out in the dialect similarity matrix in the i-th round, denotes the set of all neighbor regions screened out in the dialect similarity matrix .

[0149] It can also be expressed as:

[0150]

[0151] In an embodiment, the dynamic threshold​​ It can be a fixed value set in advance according to the actual scene, which can be changed according to different scenes, for example, dynamic threshold It is set to 85%.

[0152] Dynamic threshold It is preferably self-adapted according to the similarity distribution to avoid too high threshold leading to failure of fusion or too low threshold leading to introduction of noise data.

[0153] Dynamic threshold The calculation formula is:

[0154]

[0155]

[0156]

[0157]

[0158] Among them, represents the mean of the identification accuracy of all neighboring areas of area i, represents the standard deviation of the identification accuracy of all neighboring areas of area i, represents an adjustable parameter, represents a preset minimum threshold, represents the set of all neighboring areas filtered out in the dialect similarity matrix in the first round, represents the set of all neighboring areas filtered out in the dialect similarity matrix in the first round. Dynamic threshold

[0159] The calculation formula is preferably:

[0160]

[0161]

[0162]

[0163]

[0164] Among them, represents the mean of the identification accuracy of all neighboring areas of area i, represents the standard deviation of the identification accuracy of all neighboring areas of area i, represents a preset minimum threshold, represents the set of all neighboring areas filtered out in the dialect similarity matrix in the first round, ​All the neighbor region sets screened out in the middle, representing the dialect similarity matrix All the neighbor region sets screened out in the middle.

[0165] In the dynamic threshold calculation formula of the above two different ways, is a preset value, which can be changed according to the actual scene, for example Set the position to 75%.

[0166] In an embodiment, in step S3, the enhanced region data set is subjected to sample balancing processing. If the number of dialect speech samples in any region data set is greater than a preset number, part of the dialect speech samples are randomly deleted until the number of dialect speech samples in the region data set is not greater than the preset number.

[0167] In this embodiment, the enhanced region data set after fusion can be judged to determine whether the number of dialect speech samples after fusion is greater than a preset number. If it is greater, sample balancing processing needs to be performed to avoid training bias caused by too large number of samples in a region.

[0168] S4, student model training: using the enhanced region data set of each region to train the respective student model, repeating steps S3 to S4 until round fusion training is performed, and the student model of each region is trained to completion.

[0169] Each region has its own independent student model. There are several regions, and there are several student models. Each region independently trains its own student model using the region data set of each region after fusion with neighbor regions, and obtains the trained student model.

[0170] In the present application, not only one data fusion is performed, but multiple rounds (K times) of data fusion are performed. After multiple fusions, the ideal state is that the enhanced region data set has K region data sets fused together. When repeating step S3, the enhanced region data set is used to perform recognition on the student model itself and other student models, and the recognition accuracy is obtained by comparing the two sets of recognition results.

[0171] The present application designs a multi-round progressive training process. After each round of training, the similarity matrix and the enhanced data set are dynamically updated. Combined with the "data distillation + KNN dynamic data fusion" strategy, the training data distribution and model capability of each region student model are gradually optimized, and the modeling and generalization capability of the model for regional dialect acoustics and language characteristics is significantly enhanced.

[0172] In an embodiment, the student model adopts a model with the same Conformer-CTC structure as the student model.​

[0173] In an embodiment, unlike the sub-model, the loss function of the student model adopts a multi-objective joint distillation loss function, which is defined as:

[0174]

[0175] wherein, represents the KL divergence loss between the enhanced regional data set in the preset teacher model prediction distribution and the student model output distribution U, which is used to measure the knowledge distillation effect. is the CTC loss, which is the connectionist temporal classification loss for the end-to-end speech transcription task. is the cross-entropy loss, which calculates the consistency between the student model prediction and the pseudo label. is a weight hyperparameter, which is used to adjust the contribution of each loss in the total loss.

[0176] Through the above multi-objective joint loss, the student model not only learns the soft label information of the teacher model, but also considers the end-to-end sequence prediction ability and label consistency, thereby improving the model performance. Through multi-round training combined with multi-objective joint distillation training, the challenges brought by dialect data scarcity and diversity are effectively solved, and the performance and practical value of dialect speech recognition are significantly improved.

[0177] In the present embodiment, , and The calculation formulas of and are prior art and will not be repeated here.

[0178] In an embodiment, the preset teacher model is a random one or the one with the best performance among the several teacher models with different structures in step S1, wherein the best performance refers to the one that contributes the most pseudo labels in step S1.

[0179] In an embodiment, the preset teacher model is a SenseVoice Small model.

[0180] The present embodiment also provides a dialect speech recognition system, which comprises:

[0181] A data processing module is configured to obtain regional data sets of several regions, wherein the regional data sets comprise a plurality of unlabeled dialect speech samples, and the data processing module is configured to preprocess all the dialect speech samples, wherein the preprocessing comprises one or a combination of a plurality of combinations of the following: speech length filtering, speech activity detection segmentation, audio denoising, volume normalization, and feature extraction.

[0182] The multi-teacher model pseudo-label generation module is configured to use a plurality of teacher models with different structures to infer all dialect speech samples to obtain a plurality of candidate labels corresponding to the dialect speech samples;

[0183] The large language model screening module is configured to screen pseudo-labels corresponding to the dialect speech samples from the plurality of candidate labels to obtain a plurality of regional data sets each including a plurality of dialect speech samples with pseudo-labels;

[0184] The sub-model initialization module is configured to use the regional data set of each region to initialize and train a sub-model of the region to obtain a trained sub-model of each region;

[0185] The dialect similarity calculation module is configured to calculate the recognition accuracy of each sub-model of each region on the regional data set of another region to obtain a dialect similarity matrix;

[0186] The K-nearest neighbor dynamic data fusion module is configured to select, in the dialect similarity matrix, one other region with the highest recognition accuracy of the current region as a neighbor region, and combine the regional data set corresponding to the neighbor region into the regional data set of the current region to form an enhanced regional data set;

[0187] The joint distillation training module is configured to use the enhanced regional data set of each region to train a student model of the region, and perform multi-round dynamic training using the dialect similarity calculation module and the K-nearest neighbor dynamic data fusion module until rounds of fusion training are performed to obtain a trained student model of each region.

[0188] The embodiment of the present application also provides a dialect speech recognition model, which adopts the student model obtained by the dialect speech recognition method provided in each of the above embodiments of the present application.

[0189] Embodiment 1

[0190] With reference to Figures 2 to 3 , the present embodiment provides a dialect speech recognition method, which includes the following steps:

[0191] S1, pseudo-label generation and screening

[0192] (1) Data preprocessing

[0193] The dialect voice samples collected from 28 counties and cities in Guizhou Province are preprocessed to ensure the stability and reliability of the data input into the model. The audio data of each county and city is about 20 hours, with a sampling rate of 16000Hz. The corpus types include daily conversations, news broadcasts, and local government texts. Audio data with a length less than 0.5 seconds or more than 20 seconds is filtered out. The audio data is divided by VAD, volume normalized (target decibel value is -20dB), noise suppressed to generate high signal-to-noise ratio speech slices, and 80-dimensional Mel spectrum features are extracted. A unigram word segmentation model is used to build a word list with a size of 5000.

[0194] 28 Each county and city has its own independent Guizhou dialect unannotated dataset as a regional dataset, which contains multiple preprocessed dialect voice samples.

[0195] (2) Multi-teacher model generates pseudo labels

[0196] Three different teacher models, Paraformer model, DeepSpeech2 model, and SenseVoice Small model (i.e. SenseVoice model), are used to infer all Guizhou dialect unannotated datasets to obtain multiple candidate labels corresponding to the dialect voice samples, forming a multi-label candidate dataset.

[0197] Specifically, the three different teacher models are denoted as T1, T2, and T3. For the mth dialect voice sample , the multi-label candidate dataset is generated as follows:

[0198]

[0199] Where represents the recognition result of the sample by the teacher model T1, represents the recognition result of the sample by the teacher model T2, represents the recognition result of the sample by the teacher model T3.

[0200] (3) Large model filters pseudo labels

[0201] The Qwen2.5-72B large language model is used to filter the pseudo labels corresponding to each dialect voice sample through pre-set prompt words, and finally a high-confidence pseudo label dataset is obtained.

[0202] Specifically, the following is an example process of filtering with the Qwen2.5-72B model:

[0203] Input preparation:

[0204] Assuming that three candidate labels are generated for a dialectal speech sample by three teacher models, they are:

[0205] Label 1: "I went to the market to buy vegetables today."

[0206] Label 2: "I went to the market to buy vegetables today."

[0207] Label 3: "I went to the market to buy vegetables today."

[0208] System prompt design:

[0209] System prompt:

[0210] You are a language expert trained on a Guizhou dialect speech recognition dataset. Your task is:

[0211] (1) Select the most consistent, semantically reasonable, and syllable-compliant sentence with the Guizhou dialect daily expression from the candidate sentences.

[0212] (2) Please only output the final selected sentence, without any explanation.

[0213] (3) The candidate sentences are as follows.

[0214] User input:

[0215] Candidate sentence 1: "I went to the market to buy vegetables today."

[0216] Candidate sentence 2: "I went to the market to buy vegetables today."

[0217] Candidate sentence 3: "I went to the market to buy vegetables today."

[0218] Please select the most appropriate sentence according to the Guizhou dialect context and output the final selected sentence.

[0219] Model output:

[0220] I went to the market to buy vegetables today.

[0221] Result processing:

[0222] Save the output result as the pseudo-label of the dialectal speech sample.

[0223] In actual implementation, you can configure system roles and role inputs according to actual API interfaces.

[0224] S2, sub-model initialization training

[0225] The sub-models for each of the 28 counties and cities in Guizhou Province were initialized and trained, with the training objective being to minimize the CTC loss function. The sub-models adopted a Conformer-CTC structure. Each sub-model includes an encoder and a decoder. The encoder consists of a multi-head self-attention module, a convolutional module, a feedforward network module, and a layer normalization and residual connection module connected in sequence. The decoder uses a Connectionized Temporal Classification (CTC) decoding layer. The input to the sub-model is either dialect speech samples or pre-processed dialect speech samples, and the output is the transcribed text corresponding to the dialect speech samples.

[0226] S3, Data Fusion

[0227] (1) KNN dynamic data fusion

[0228] A dialect similarity matrix was constructed using cross-validation:

[0229]

[0230] Right now:

[0231] in, This represents the regional dataset of the i-th county / city. This represents the sub-model of the j-th county / city. Submodel Regional datasets Recognition accuracy .

[0232] In the dialect similarity matrix, select the region dataset. The neighboring region with the highest accuracy is identified and compared with a dynamic threshold. If the accuracy is not less than a threshold, the corresponding region dataset is merged into the new region dataset. If an augmented dataset is obtained, the merging process is abandoned; otherwise, the dataset is discarded. The dynamic threshold is adaptively adjusted based on the similarity distribution.

[0233] S4, Multi-objective Joint Loss Training

[0234] Each student model, which uses the same Conformer-CTC structure as the sub-model, is trained on its corresponding augmented dataset. The student model uses a multi-objective joint distillation loss function.

[0235] The default teacher model is the SenseVoice Small model.

[0236] Determine which round, steps S3 and S4, it is; if not... If the cycle continues, then steps S3 and S4 will be repeated until the desired result is reached. The end condition of the wheel finally completes the training of the student model, and outputs 28 student models for use in the dialect phonetic transcription of 28 counties and cities in Guizhou.

[0237] The embodiment is performed on an NVIDIA RTX 3090 GPU with 24 GB of video memory, the operating system is Windows 10, the development language is Python 3, and the deep learning framework is PyTorch.

[0238] During the training of the sub-model, the maximum number of training rounds is set to 50 rounds, the batch size is set to 16, the gradient accumulation (accum_grad) is set to 4, the optimizer is Adam, the initial learning rate is 0.001, the weight decay is 1e-6, the learning rate scheduling strategy is WarmupLR, the warmup step number is 25000, the minimum learning rate is 1e-5, and the gradient clipping threshold is set to 5.0.

[0239] In terms of model structure, the encoder adopts the Conformer architecture, the output dimension is 256, the number of Attention heads is 4, the position feedforward layer dimension is 2048, the number of encoder blocks is 12, the kernel size of the convolution module is 15, the swish activation function is used, the position encoding is relative position encoding (rel_pos), the input layer type is conv2d, and the dropout rate is 0.1. The decoder adopts the BiTransformerDecoder structure, the number of Attention heads is 4, the feedforward layer dimension is 1024, the number of decoder blocks and reverse decoder blocks is 3, and the dropout rate is 0.1.

[0240] The results show that after the multi-teacher distillation and KNN dynamic data fusion combined training are completed, the average CER of each student model on the test set of each county and city reaches 95%, which greatly leads the recognition effect of the mainstream dialect model, verifying the effectiveness and superiority of the method in the complex multi-dialect phonetic transcription task. It can be seen that, compared with other dialect speech recognition algorithms using traditional RNN or CTC architecture, the present application shows significant improvement in transcription accuracy.

[0241] The preferred embodiments of the present application have been described in detail above, but it should be understood that, after reading the above description of the present application, those skilled in the art can make various modifications or changes to the present application. These equivalent forms also fall within the scope defined by the claims attached hereto.

Claims

1. A dialect speech recognition method, characterized in that, The dialect speech recognition method includes: S1. Obtain regional datasets for several regions. The regional datasets include several unlabeled dialect speech samples. Use several teacher models with different structures to infer all the dialect speech samples to obtain multiple candidate labels corresponding to the dialect speech samples. Filter out the pseudo labels corresponding to the dialect speech samples from the multiple candidate labels to obtain several regional datasets including several dialect speech samples with pseudo labels. S2, use the regional datasets of each region to initialize and train their respective sub-models, and obtain the trained sub-models for each region; S3, calculate the recognition accuracy of the sub-model of each region on the other region datasets to obtain the dialect similarity matrix. In the dialect similarity matrix, select the other region with the highest recognition accuracy of the current region as the neighboring region, and merge the regional datasets corresponding to the neighboring region into the regional dataset of the current region to form an enhanced regional dataset. S4, train the respective student models using the enhanced regional datasets for each region, repeating steps S3 to S4 until... Through rounds of fusion training, student models trained in each region are obtained; In step S3, cross-validation is used to process the regional dataset of region i. In the sub-model of region j Identification is performed on the above, among which , obtain the transcribed text The region dataset of region i In its corresponding sub-model The written text obtained by recognition With transcribed text By comparing, sub-models are obtained. Regional datasets Recognition accuracy ,in, The formula for calculating the recognition accuracy is: ； ； ； WER stands for Written Text. With transcribed text The word error rate is defined as follows: S1 represents the number of words replaced incorrectly, D1 represents the number of words deleted incorrectly, I1 represents the number of words inserted incorrectly, and N1 represents the total number of words in the reference text; CER represents the written text. With transcribed text The error rate is calculated as follows: S2 represents the number of characters replaced incorrectly, D2 represents the number of characters deleted incorrectly, I2 represents the number of characters inserted incorrectly, and T represents the total number of characters in the reference text. All cross-validation results constitute Dialect similarity matrix ,but: ； For region i, in the dialect similarity matrix In the dataset, the region j with the highest recognition accuracy (excluding itself) is selected as the neighboring region, and the corresponding region datasets are then processed. Merging into regional datasets In this process, an enhanced regional dataset is formed. ; In the dialect similarity matrix In the middle, Elements with similarity below the preset threshold are set to zero; In step S3, after selecting the other region with the highest recognition accuracy as the neighboring region, before merging the regional dataset corresponding to the neighboring region into the regional dataset of the current region, the method further includes: determining whether the selected highest recognition accuracy is not less than a preset dynamic threshold. If it is not less than the threshold, the regional dataset corresponding to the neighboring region is merged into the regional dataset of the current region; otherwise, the merging is abandoned. In step S3, the enhanced regional dataset is subjected to sample balancing processing. The sample balancing processing method is as follows: if the number of dialect speech samples in any regional dataset is greater than a preset number, some dialect speech samples are randomly deleted until the number of dialect speech samples in the regional dataset is not greater than the preset number.

2. The dialect speech recognition method as described in claim 1, characterized in that, In step S1, after obtaining regional datasets for several regions, before using several teacher models with different structures to infer all the dialect speech samples, the method further includes: preprocessing all the dialect speech samples, wherein the preprocessing includes one or more combinations of speech length filtering, speech activity detection segmentation, audio denoising, volume normalization, and feature extraction. And / or, in step S1, three teacher models with different structures are used to infer all the dialect speech samples. The three teacher models are Paraformer model, DeepSpeech2 model and SenseVoice Small model. And / or, in step S1, filtering out pseudo-labels corresponding to the dialect speech samples from a plurality of candidate labels includes: inputting a plurality of candidate labels corresponding to each dialect speech sample into a preset large language model, and filtering by preset prompt words in the large language model so that the large language model filters out pseudo-labels corresponding to each dialect speech sample; And / or, in step S1, several regions are different counties or cities within the same province, different administrative regions within the same municipality, or different cities within the same autonomous region.

3. The dialect speech recognition method as described in claim 2, characterized in that, When filtering speech length, dialect speech samples shorter than 0.5 seconds and longer than 20 seconds are removed; And / or, the target volume is -20dB when performing volume normalization; And / or, when performing feature extraction, fbank feature extraction based on an 80-Vimel spectrum filter bank is used; And / or, the prompt word includes one or more of the following combinations: dialect context consistency judgment, semantic rationality detection, syllable matching degree analysis, and regional geographic location information.

4. The dialect speech recognition method as described in claim 1, characterized in that, In step S2, the regional datasets of each region and a portion of the manually labeled datasets of each region are used to initialize and train the respective sub-models; And / or, in step S2, the sub-model adopts a Conformer-CTC structure; the sub-model includes an encoder and a decoder, the encoder includes a multi-head self-attention module, a convolution module, a feedforward network module, and a layer normalization and residual connection module connected in sequence, and the decoder adopts a connection-based temporal classification decoding layer; the model input of the sub-model is a dialect speech sample or a preprocessed dialect speech sample, and the model output of the sub-model is the transcribed text corresponding to the dialect speech sample; And / or, in step S2, the training objective during the initial training of the sub-model is to minimize the CTC loss function; And / or, in step S2, when initializing the training of the sub-model, one or more combined strategies of pre-trained model loading, optimizer configuration, and parallel training are adopted; the pre-trained model loading strategy is to load pre-trained weights on the Mandarin speech recognition dataset when initializing the sub-model; the Mandarin speech recognition dataset is at least one of AISHELL-1 and WeTSpeech; the optimizer configuration strategy is to use the Adam optimizer and combine it with SpecAugment for data augmentation; the initial learning rate of the Adam optimizer is 1e-4, and the warm-up ratio is 0.1; the parallel training strategy is that the sub-models corresponding to each region are trained in parallel on a multi-GPU cluster.

5. The dialect speech recognition method as described in claim 1, characterized in that, The dynamic threshold The calculation formula is: ； ； ； ； in, This represents the average recognition accuracy of all neighboring regions of region i. This represents the standard deviation of the identification accuracy of all neighboring regions of region i. This indicates an adjustable parameter. This represents the preset minimum threshold. Indicates the first Round-robin in dialect similarity matrix The set of all neighboring regions selected in the middle, In the dialect similarity matrix The set of all neighboring regions selected in the middle; Or, the dynamic threshold The calculation formula is: ； ； ； ； in, This represents the average recognition accuracy of all neighboring regions of region i. This represents the standard deviation of the identification accuracy of all neighboring regions of region i. This represents the preset minimum threshold. Indicates the first Round-robin in dialect similarity matrix The set of all neighboring regions selected in the middle, In the dialect similarity matrix The set of all neighboring regions selected from the data.

6. The dialect speech recognition method according to any one of claims 1 to 4, characterized in that, The student model adopts the same Conformer-CTC structure as the sub-model. The student model includes an encoder and a decoder. The encoder includes a multi-head self-attention module, a convolution module, a feedforward network module, and a layer normalization and residual connection module connected in sequence. The decoder adopts a connected temporal classification decoding layer. The model input of the student model is a dialect speech sample or a preprocessed dialect speech sample. The model output of the student model is the transcribed text corresponding to the dialect speech sample. And / or, the loss function of the student model adopts a multi-objective joint distillation loss function, which is defined as: ； in, This indicates the distribution of the enhanced regional dataset in the pre-defined teacher model prediction. The KL divergence loss between the student model output distribution U and the KL divergence loss. For CTC losses, For cross-entropy loss, These are weight hyperparameters; The preset teacher model is either a random one from several teacher models with different structures in step S1 or the one with the best performance, where the best performance means the teacher model that contributes the most pseudo-labels in step S1; or, the preset teacher model is the SenseVoice Small model.

7. A dialect speech recognition system, characterized in that, The dialect speech recognition system employs the dialect speech recognition method according to any one of claims 1 to 6, comprising: The data processing module is used to acquire regional datasets for several regions. The regional datasets include several unlabeled dialect speech samples. All dialect speech samples are preprocessed. The preprocessing includes one or more of the following combinations: speech length filtering, speech activity detection and segmentation, audio denoising, volume normalization, and feature extraction. The multi-teacher model pseudo-label generation module is used to infer all the dialect speech samples using several teacher models with different structures to obtain multiple candidate labels corresponding to the dialect speech samples. The large language model filtering module is used to filter out the pseudo-labels corresponding to the dialect speech samples from multiple candidate labels, and obtain several regional datasets including several dialect speech samples with pseudo-labels. The sub-model initialization module is used to initialize and train the respective sub-models using the regional datasets of each region, thereby obtaining the trained sub-models for each region. The dialect similarity calculation module is used to calculate the recognition accuracy of each region's sub-model on other region datasets, and obtain the dialect similarity matrix; The K-nearest neighbor dynamic data fusion module is used to select the other region with the highest recognition accuracy to the current region from the dialect similarity matrix as the neighbor region, and merge the regional datasets corresponding to the neighbor region into the regional dataset of the current region to form an enhanced regional dataset. The joint distillation training module is used to train the respective student models using the regionally enhanced datasets from each region. It performs multiple rounds of dynamic training using the dialect similarity calculation module and the K-nearest neighbor dynamic data fusion module, until... Through rounds of fusion training, student models trained in each region are obtained.

8. A dialect speech recognition model, characterized in that, The dialect speech recognition model is a student model obtained using the dialect speech recognition method described in any one of claims 1 to 6.

Images