A dialect speech recognition method, system and handheld terminal

Abstract

This invention relates to a dialect speech recognition method, system, and handheld terminal. It extracts multi-dimensional acoustic features from the speech waveform of a target dialect, constructs an acoustic feature sequence, employs unsupervised pre-training, and learns the inherent linguistic structure and feature associations of the target dialect by masking part of the acoustic feature sequence and utilizing contextual association prediction, thus obtaining a pre-trained model. The target dialect is collected and labeled to obtain labeled corpus. A general language model is then grafted onto the pre-trained model, some low-level parameters of the pre-trained model are frozen, and the general language model is adaptively adjusted using the labeled corpus to obtain a dialect speech recognition model for dialect speech recognition. The method enables the implementation of a dialect speech recognition system and a handheld terminal for dialect speech recognition. This invention significantly reduces upfront costs and technical barriers, significantly improves recognition accuracy, achieves a complete transformation from feature learning to practical recognition, adapts to various dialects with different linguistic features, and has a wide range of applications.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing, and in particular to a dialect speech recognition method, system, and handheld terminal. Background Technology

[0002] From a historical and linguistic perspective, dialect systems are vast and complex. Most dialects have evolved over thousands of years, forming unique linguistic features distinct from Standard Mandarin and other languages. Taking Taizhou dialect as an example, it originated from the Taihu sub-dialect of Wu dialect and underwent fusion with ancient Yue dialect and Central Plains Mandarin, forming a unique seven-tone system, rich nasalized rhymes, and special initial and final combinations. Furthermore, subtle prosodic differences exist between dialects in different districts and counties, resulting in a highly distinctive linguistic structure. In terms of the overall state of dialects, most lack a unified standardized writing system, exhibit complex phonetic evolution patterns, and flexible colloquial expressions, showing significant differences from Standard Mandarin in tone, phonemes, and prosody.

[0003] Currently, dialect speech recognition technology generally faces core bottlenecks. On the one hand, existing technologies largely rely on large amounts of manually annotated corpora and dedicated phoneme dictionaries. However, dialect annotation requires the participation of professional linguistic researchers, which is not only time-consuming and labor-intensive, but also extremely costly and time-consuming, making it difficult to achieve large-scale promotion and application. This is especially true for niche dialects, where the scarcity of annotated corpora is even more pronounced. On the other hand, existing general-purpose speech recognition models are mostly trained on Mandarin or mainstream languages, failing to fully adapt to the unique linguistic features of various dialects. They cannot accurately capture the unique tonal changes, special phonemes, and prosodic rules of dialects, resulting in low dialect recognition accuracy and poor adaptability, making it difficult to meet practical application needs.

[0004] Unsupervised pre-trained models have gradually matured in the field of speech feature learning. However, most existing unsupervised models are designed for general speech or a single language, lacking a deep integration with dialect linguistic features and making it difficult to accurately uncover the inherent speech structure relationships of dialects. Even if they can initially learn the basic features of dialects, they cannot be directly adapted to downstream dialect speech recognition tasks. There is a lack of a targeted connection scheme between "feature learning and task adaptation." Existing fine-tuning techniques mostly follow the fine-tuning logic of general speech recognition and do not optimize fine-tuning strategies in combination with the linguistic characteristics of dialects. This results in insufficient dialect adaptability of the fine-tuned model and fails to fully leverage the advantages of unsupervised pre-training.

[0005] In summary, there is an urgent need for a low-cost, highly adaptable, and scalable dialect speech recognition technology to break through the current technological bottleneck. Summary of the Invention

[0006] This invention solves the problems existing in the prior art and provides a dialect speech recognition method, system and handheld terminal.

[0007] The technical solution adopted in this invention is a dialect speech recognition method, which extracts multi-dimensional acoustic features from the speech waveform of the target dialect, constructs an acoustic feature sequence, adopts an unsupervised pre-training method, and learns the inherent linguistic structure and feature association of the target dialect by masking part of the acoustic feature sequence and using context association to predict the masked part, thereby obtaining a pre-trained model.

[0008] Several target dialects were collected and labeled to obtain labeled corpora. A general language model was grafted onto the pre-trained model, some of the underlying parameters of the pre-trained model were frozen, and the general language model was adaptively adjusted using the labeled corpora to obtain a dialect speech recognition model for dialect speech recognition.

[0009] Preferably, the multidimensional acoustic features include at least one of basic acoustic features, tone features, special phoneme features, and prosodic features;

[0010] The multidimensional acoustic features are quantified based on syllable-level linguistic units, and each syllable-level linguistic unit includes at least:

[0011] The first set of parameters used to characterize the differences in phonetic segments of the target dialect;

[0012] The second parameter set used to characterize the differences in tone and phonological changes in the target dialect;

[0013] The third parameter set used to characterize the continuous prosodic features of the target dialect;

[0014] The third parameter set includes several continuous normalized values ​​that reflect syllable duration, fundamental frequency dynamics, and energy distribution.

[0015] Preferably, in unsupervised pre-training, masking is performed according to dialect feature modules, wherein the dialect feature modules include at least one of tone modules, special phoneme modules, and prosody modules, wherein each dialect feature module corresponds to the acoustic feature corresponding to the masking.

[0016] Preferably, the masking probability in unsupervised pre-training is dynamically determined based on the dialect linguistic saliency of the syllable-level linguistic unit; the dialect linguistic saliency is determined by weighted average of the dialect distinguishing values ​​of the syllable-level linguistic unit at the segment level, tone level, and speech flow prosody level.

[0017] The masking operation employs multi-level masking granularity, including at least fine-grained masking for single-segment units and prosodic span masking for multiple consecutive syllables;

[0018] For linguistic units with high presupposition value, reduce the probability of them being obscured.

[0019] Preferably, the masked content includes continuous segments or discrete points in the acoustic feature sequence; the proportion of the masked part to the total length of the acoustic feature sequence is dynamically adjusted according to the linguistic complexity of the target dialect.

[0020] Preferably, a joint loss function for unsupervised pre-training is constructed, wherein the joint loss function includes several sub-loss terms constrained by the linguistic characteristics of the target dialect, and the weighted sum of the sub-loss terms constitutes the total loss.

[0021] Preferably, the labeled corpus is obtained by preprocessing the corpus of the target dialect, the preprocessing including noise removal, uniform sampling rate, and text annotation.

[0022] Preferably, some of the underlying parameters of the frozen pre-trained model are parameter layers that retain the learning results of dialect features.

[0023] A dialect speech recognition system, comprising:

[0024] The feature extraction module is used to extract multidimensional acoustic features from the speech waveform of the target dialect and construct an acoustic feature sequence.

[0025] The unsupervised pre-training module is used to learn the inherent linguistic structure and feature associations of the target dialect by masking the contents of a partial acoustic feature sequence and using contextual associations to predict the masked part, thereby obtaining a pre-trained model.

[0026] The adjustment module is used to graft a general language model onto the pre-trained model, freeze some of the underlying parameters of the pre-trained model, and adaptively adjust the general language model using labeled corpora to obtain a dialect speech recognition model.

[0027] A dialect speech recognition handheld terminal, comprising:

[0028] The acquisition unit is used to acquire dialect speech from users;

[0029] Display unit; and

[0030] The controller, coupled to the acquisition unit and the display unit, is used to perform the dialect speech recognition method on the dialect speech, or to input the dialect speech into the dialect speech recognition system to obtain the corresponding text recognition result; and to control the display unit to display the text recognition result.

[0031] This invention relates to a dialect speech recognition method, system, and handheld terminal. The method extracts multi-dimensional acoustic features from the speech waveform of a target dialect, constructs an acoustic feature sequence, and employs an unsupervised pre-training approach. By masking part of the acoustic feature sequence and using contextual association to predict the masked portion, the inherent linguistic structure and feature associations of the target dialect are learned, resulting in a pre-trained model. Several target dialects are collected and annotated to obtain an annotated corpus. A general language model is then grafted onto the pre-trained model, some low-level parameters of the pre-trained model are frozen, and the general language model is adaptively adjusted using the annotated corpus to obtain a dialect speech recognition model for dialect speech recognition. The method enables the implementation of a dialect speech recognition system and a dialect speech recognition handheld terminal.

[0032] The beneficial effects of this invention are as follows:

[0033] (1) Unsupervised training does not rely on labeled corpora and phoneme dictionaries. It can learn dialect features autonomously by relying solely on the contextual association of feature sequences, which greatly reduces the initial cost and technical threshold. Subsequent fine-tuning relies on high-quality labeled corpora to improve accuracy, enabling large-scale application with controllable costs and scalability.

[0034] (2) Feature construction fully incorporates the characteristics of dialect linguistics, and targeted training enables the model to accurately capture the tone, special phonemes and prosodic features of different dialects, solving the problem of poor adaptability of existing models and significantly improving recognition accuracy.

[0035] (3) Combining the preprocessing of folk language data with targeted grafting and fine-tuning strategies, the connection barrier between unsupervised models and downstream recognition tasks is broken. Unlike the general fine-tuning logic, the results of unsupervised training are retained, and the complete transformation from feature learning to actual recognition is achieved.

[0036] (4) Unsupervised training combined with diverse folk language data can be adapted to the same dialect in different regions and accents; the construction of generalized features and training logic can be flexibly adjusted to adapt to various dialects with different linguistic features, and has a wide range of applications.

[0037] (5) Breaking away from the traditional dialect recognition model of applying general models and forcibly establishing rules, it deeply integrates dialect linguistics knowledge with unsupervised training and targeted fine-tuning, filling the technological gap of low cost, high adaptability and scalability. Attached Figure Description

[0038] Figure 1 This is a flowchart of the method of the present invention;

[0039] Figure 2 This is a schematic diagram illustrating the specific process of implementing the present invention;

[0040] Figure 3 This is a schematic diagram of the dual-branch modification of the input layer in this invention;

[0041] Figure 4 This is a schematic diagram of the system structure of the present invention;

[0042] Figure 5 This is a schematic diagram of the module structure of the handheld terminal of the present invention. Detailed Implementation

[0043] The present invention will be further described in detail below with reference to embodiments, but the scope of protection of the present invention is not limited thereto.

[0044] This invention relates to a dialect speech recognition method, which extracts multidimensional acoustic features from the speech waveform of the target dialect, constructs an acoustic feature sequence, and adopts an unsupervised pre-training method. By masking part of the acoustic feature sequence and using contextual association to predict the masked part, the inherent linguistic structure and feature association of the target dialect are learned to obtain a pre-trained model.

[0045] Several target dialects were collected and labeled to obtain labeled corpora. A general language model was grafted onto the pre-trained model, some of the underlying parameters of the pre-trained model were frozen, and the general language model was adaptively adjusted using the labeled corpora to obtain a dialect speech recognition model for dialect speech recognition.

[0046] The method of the present invention will be described in detail below.

[0047] (1) Model Architecture

[0048] In this invention, the basic architecture of the pre-trained model is Hubert Large, and the downstream recognition model adopts the Paraformer Large architecture for grafting and fine-tuning. Specifically, the original output dimension of Hubert Large is (T, 1024), where T is the number of time frames. This invention adds a linear mapping layer Linear(1024, 768) at its output to adapt the feature dimension to 768 dimensions for use by the downstream model. The input dimension of the Paraformer Large Encoder is 768 dimensions, the number of attention heads is 8, and the number of encoder layers is 50. This architecture selection is only a preferred implementation of this invention. Those skilled in the art can replace it with other Transformer-based pre-trained models (such as wav2vec 2.0, Conformer, etc.) and downstream recognition models (such as CTC / Attention hybrid models) according to actual needs, all of which are within the protection scope of this invention.

[0049] A general language model refers to a language model pre-trained based on a mainstream language (such as Mandarin or English), including but not limited to BERT based on the Transformer architecture, sequence models based on RNNs, or end-to-end models based on CTC. During the fine-tuning stage, it is grafted onto the pre-trained model and is responsible for decoding acoustic feature sequences into text sequences.

[0050] (2) Feature extraction

[0051] The multidimensional acoustic features include at least one of the following: basic acoustic features, tone features, special phoneme features, and prosodic features;

[0052] The multidimensional acoustic features are quantified based on syllable-level linguistic units, and each syllable-level linguistic unit includes at least:

[0053] The first set of parameters used to characterize the differences in phonetic segments of the target dialect;

[0054] The second parameter set used to characterize the differences in tone and phonological changes in the target dialect;

[0055] The third parameter set used to characterize the continuous prosodic features of the target dialect;

[0056] The third parameter set includes several continuous normalized values ​​that reflect syllable duration, fundamental frequency dynamics, and energy distribution.

[0057] In this invention, the basic acoustic features are general acoustic parameters extracted from speech waveforms, including but not limited to Fbank (filter bank) features, MFCC features, PLP features, or combinations thereof; in this embodiment, 80-dimensional Fbank features are used as the basic representation, and the front-end parameters are a sampling rate of 16kHz, a frame length of 25ms, and a frame shift of 10ms.

[0058] Tonal features refer to parameters used to characterize the tonal system of a dialect, including but not limited to the fundamental frequency F0 and its derived features, such as the fundamental frequency curve, tone value, tone type, and continuous tone sandhi patterns. For different dialects, the tonal features can be adjusted according to the number of tones, such as the Taizhou dialect having 7 tones (8 after tone sandhi).

[0059] Special phoneme features refer to the acoustic parameters corresponding to the unique phonemes that distinguish dialects from Mandarin or the common language, such as the acoustic quantification values ​​of nasalized finals, ancient voiced initials, special finals, glottal stop codas, and palatalized sounds in Taizhou dialect.

[0060] Prosodic features are the core components of suprasegmental features of speech, mainly used to reflect the rhythm, tone and overall prosodic style of speech, including but not limited to duration, energy, fundamental frequency (F0) and its envelope, prosodic stress, boundary tone, pauses, rhythm, etc.

[0061] The aforementioned features are constructed using methods including but not limited to concatenation, weighted summation, and / or neural network fusion to create a unified feature sequence, resulting in a 128-dimensional linguistic acoustic feature sequence that enables the quantitative transformation of dialect linguistic knowledge. Each frame of the feature sequence corresponds to a short-time window (25ms) of the speech waveform, and the sequence length depends on the duration of the input speech. The HuBERT alignment step size is 20ms.

[0062] To further enhance the model's ability to model dialect linguistic features, this invention segments the speech signal into syllable-level linguistic units, each syllable unit being denoted as... , ,in, These represent initial consonant category, final vowel category, tone category, connected speech sound change category, prosodic boundary / stress, dialect / accent category, and continuous linguistic features, respectively. Their value range varies depending on the dialect. Generally, there are 30 to 50 categories. There are generally 50 to 70 categories, which will not be listed here, and can be further subdivided according to the development of linguistics.

[0063] The embedding dimensions of each discrete parameter are fixed as follows: initial consonant 64-dimensional, final vowel 64-dimensional, tone 32-dimensional, sound change 32-dimensional, prosody 16-dimensional, and dialect category 16-dimensional. Among these, continuous linguistic features... A 6-dimensional vector, represented as ;

[0064] This is a syllable duration parameter used to distinguish dialectal phenomena such as neutral tone, residual entering tone, and weakened retroflex endings, quantifying the relative differences in syllable duration; let syllable... The actual duration is The speaker's global median syllable duration is Calculate the ratio of calculation time After truncation and normalization The minimum syllable duration is 80ms, the maximum syllable duration is 350ms, and the default median duration is 180ms. It indicates that the syllable is significantly shorter (such as entering tone or neutral tone). This indicates that the syllable is significantly longer (such as stressed syllables or prolonged sounds).

[0065] This is the fundamental frequency mean parameter, used to reflect tone category differences and the speaker's normalized pitch level, suitable for the differences between dialect tone values ​​and Standard Mandarin. Let the fundamental frequency mean of the sound segments within a syllable be... The speaker's logarithmic fundamental frequency mean and standard deviation are respectively Calculate the standardized value: sigmoid compressed to [0,1]: ,in The higher the final value, the higher the pitch of the syllable relative to the speaker, which can be used to distinguish between dialectal high-level tones, rising tones, and short entering tones, etc.

[0066] The fundamental frequency slope parameter is used to characterize the tone trend (flat, rising, falling, and turning), suitable for complex tone sandhi patterns in dialects; a linear fit is performed on the F0 curve within the syllable to obtain the slope. Normalization process yields Preferred (Unit: semitone / 100ms); To clearly lower the pitch, It is a level tone. It is a clear upward inflection;

[0067] Let be an energy parameter, representing phenomena such as stress, weakening, and neutral tone, suitable for the differences in stress distribution between dialects and Standard Mandarin; let the average syllable energy be . The average speaker value and standard deviation were respectively Calculate standardized values sigmoid is compressed to [0,1]. A high value corresponds to stressed syllables, while a low value corresponds to unstressed syllables or weakened syllables (such as unstressed syllable variants in dialects).

[0068] The preceding and following pause parameters are used to model pauses and prosodic boundaries, adapting to differences in rhythmic patterns in spoken language; let the lengths of the preceding and following pauses be respectively... and Take the larger value as the boundary strength. Normalization process yields (With a pause limit of 300ms); The higher the syllable, the more likely it is to be located at the word boundary, phrase boundary, or sentence end, conforming to the rhythmic habits of dialects;

[0069] This is a local speech rate parameter used to represent the speed of speech flow within a syllable, suitable for phenomena such as tone sandhi, weakening, and elision in dialects; a window of ±2 syllables is taken centered on the syllable, and the average syllable duration within the window is calculated. Calculate the speech rate ratio truncation normalization A high final value corresponds to a fast local speech rate (prone to connected speech and tone sandhi), while a low value corresponds to a slow local speech rate (prone to preserving complete tone values).

[0070] (3) Unsupervised pre-training and masking strategies

[0071] In the unsupervised pre-training of the present invention, masking is performed according to dialect feature modules, wherein the dialect feature modules include at least one of tone modules, special phoneme modules and prosody modules, wherein each dialect feature module corresponds to the acoustic feature corresponding to the masking.

[0072] The masking mentioned in this invention refers to replacing the values ​​of specific positions or dimensions in an acoustic feature sequence with special masking markers, such as the [MASK] symbol or zero values, while preserving the context position of the masked portion. The sequence length remains unchanged after masking, and the model uses the unmasked positions as context to predict the original values ​​of the masked positions. In practical applications, masking strategies include, but are not limited to, dialect feature module masking and / or random masking. The former masks all dimensions belonging to a certain type of feature in the entire feature sequence at once, while the latter randomly selects some positions or dimensions in the sequence for masking according to a certain probability.

[0073] The masking probability in unsupervised pre-training is dynamically determined based on the dialect linguistic saliency of the syllable-level linguistic unit; the dialect linguistic saliency is determined by weighted average of the dialect distinguishing values ​​of the syllable-level linguistic unit at the segment level, tone level, and speech flow prosody level.

[0074] The masking operation employs multi-level masking granularity, including at least fine-grained masking for single-segment units and prosodic span masking for multiple consecutive syllables;

[0075] For linguistic units with high presupposition value, reduce the probability of them being obscured.

[0076] Specifically, to improve the targeting of the masking strategy for dialect feature learning, this invention applies a method to each syllable unit. Define significance score (Normalized to [0,1]), used to measure the dialect distinguishing value of this unit; the formula and weights are fixed.

[0077]

[0078] in, For segment salience, strong dialect features (such as lateral-nasal consonant blending, preservation of palatal and velar consonants, and remnants of entering tone codas) are assigned a value of 1.0; moderate dialect features (aspiration / anesthesia weakening) are assigned a value of 0.7; common shared features are assigned a value of 0.4; and low information content segments are assigned a value of 0.2.

[0079] For tone significance, 1.0 is used for entering tone, complex tone sandhi, and special tone categories; 0.8 is used for tone categories that are significantly different from Mandarin; 0.5 is used for ordinary steady tone categories; and 0.2 is used for neutral tone and weakened tone.

[0080] The significance of sound changes is calculated as follows: 1.0 for significant sound changes (such as tone sandhi, retroflexion, and nasalization); 0.6 for slight sound changes; and 0.0 for no sound changes.

[0081] The prosodic saliency is calculated from continuous parameters and satisfies the following conditions:

[0082]

[0083] To determine the significance of dialect attribution, a value of 1.0 was set for typical dialect words / pronunciations; 0.6 was set for mixed Mandarin and dialect pronunciations; and 0.2 was set for features of standard Mandarin.

[0084] Based on this, the masking probability is dynamically determined by the saliency score. The unit masking probability (monosyllabic unit) satisfies the following:

[0085]

[0086] The minimum masking is 35% when the saliency is low, and the maximum masking is 85% when the saliency is high, focusing on the core features of the dialect;

[0087] The global masking ratio (for the entire speech segment) is determined by the global dialect density, satisfying the following conditions:

[0088]

[0089] in, , The upper limit is preset, preferably 0.80~0.90; weak dialect features ( When the global mask is approximately 50%, the dialect features are strong. When the global mask is 75%–80%, the global mask is approximately 75%–80%.

[0090] The masking granularity adopts a three-layer structure. Segment-level discrete masking accounts for about 45%, which is one linguistic unit, about 40~80ms, corresponding to initials, finals, aspiration, nasalization, etc. Syllable-level continuous masking accounts for about 40%, which is one to two syllables, about 120~400ms, corresponding to initial-final-tone combinations, retroflexion, neutral tone, etc. The remainder is prosodic-level span masking, generally two to four syllables, about 250~800ms, corresponding to tone sandhi, pauses, rhythmic patterns, etc.

[0091] Specifically, the masked content includes continuous segments in the acoustic feature sequence, such as a specified feature dimension of 5-20 consecutive frames, or discrete points, such as a randomly selected single frame or a single dimension.

[0092] Another feasible approach is to dynamically adjust the proportion of the masked portion to the total length of the acoustic feature sequence based on the linguistic complexity of the target dialect. The higher the complexity, such as a rich tone system or many special phonemes, the greater the masking proportion can be, in order to enhance the learning difficulty and generalization ability of the model.

[0093] To avoid excessive masking of core dialect features, a protection method of "reducing the masking probability" is adopted. Let the key feature protection coefficient be... Protection mask probability The preferred protection coefficients include high-frequency key features (such as dialect-specific tones and entering tones), low-frequency key features (such as the mixing of palatal and velar consonants and lateral and nasal consonants), and very low-frequency key features (such as special literary and colloquial pronunciations), which respectively correspond to The value can be 0.7, 0.5, or 0.3.

[0094] (4) Input layer dual-branch modification

[0095] To simultaneously utilize both raw acoustic signals and structured linguistic information, this invention modifies the input layer of the pre-trained model into a dual-branch design, including:

[0096] The first branch (original speech branch) is used to input the original speech waveform (16kHz sampled audio), and the basic acoustic features are extracted by the convolutional front end, preserving continuous information such as spectrum, duration, and formant changes;

[0097] The second branch (dialect quantization feature branch) takes the quantized dialect feature sequence as input and maps the discrete labels to continuous vector representations through an embedding layer. The embedding dimensions are: initial consonant (64-dimensional), final vowel (64-dimensional), tone (32-dimensional), sound change (32-dimensional), prosody (16-dimensional), and dialect category (16-dimensional). The continuous parameter G... j The two MLP layers (128-dimensional hidden layer) are projected to 32-dimensional vectors, and the total dimension after concatenation is 256-dimensional. Then, the 256-dimensional embedding vectors are mapped to the input dimension of the HuberT model (1024-dimensional for the Large model) through a linear layer. The outputs of the two branches are weighted and fused, with the optimal fusion weight being 0.6:0.4 (acoustic features: dialect quantization features).

[0098] Furthermore, this invention adds a dedicated mask embedding for dialect quantization units at the input end, with the same dimension as the HuBERT input dimension (1024 dimensions in the Large model). It is updated synchronously with the dialect feature representation during training, improving the model's ability to recover the masked linguistic units.

[0099] (5) Loss Function

[0100] Construct a joint loss function for unsupervised pre-training, wherein the joint loss function includes several sub-loss terms constrained by the linguistic characteristics of the target dialect, and the weighted sum of the sub-loss terms constitutes the total loss;

[0101] In this embodiment, the sub-loss terms include:

[0102] Masked feature prediction loss is used to constrain the model's prediction accuracy for masked linguistic parameters;

[0103] Speech-phonological consistency loss is used to constrain the consistency between the model output and the distribution of the original dialect quantized features;

[0104] Feature distance loss is used to enhance the clustering of the same sound class in the feature space;

[0105] Auxiliary temporal loss is used to help the model establish stable temporal output.

[0106] Specifically, to enable the model to better learn dialect speech features under unsupervised or weakly supervised conditions, this scheme adopts a joint loss form. Combining the quantization characteristics of dialect features, the total loss can be expressed as:

[0107]

[0108] Each loss corresponds to a different level of linguistic constraint, and the weight parameters α, β, γ, δ can be dynamically adjusted according to the training phase.

[0109] (5-1) Mask Feature Prediction Loss

[0110] The model predicts masked dialect quantization units, targeting quantized linguistic parameters such as initial consonant category, final vowel category, and tone category. Cross-entropy loss is employed.

[0111]

[0112] Where M is the number of mask units and K is the number of prediction categories, such as 40 categories for initials and 9 categories for tones. For real labels, This loss predicts probabilities for the model; compared to the original frame-level classification, it targets linguistically defined phonological categories and focuses on dialect feature learning.

[0113] (5-2) Speech-phonological consistency loss

[0114] For unmasked regions, the constraint model output maintains a consistent distribution with the original dialect quantization features, employing KL divergence loss.

[0115]

[0116] in, The distribution of quantitative features of the original dialect. The distribution of features output by the model. The number of unmasked units; this loss can reduce the drift of linguistic attributes during training and ensure that the model output features have stable dialect attributes.

[0117] (5-3) Feature distance loss

[0118] The hidden representation output by the model is subjected to distance constraints with the target quantized feature embedding vector, and Euclidean distance loss is used.

[0119]

[0120] in, The hidden representation output by the model. The embedding vector for dialect quantification features. The total number of units represents the loss function. This loss function can enhance the clustering of similar phonological classes, similar tone classes, and similar phonological changes in the feature space, thereby improving the feature discrimination.

[0121] (5-4) Auxiliary timing loss

[0122] A lightweight CTC branch is added after the top-level encoder, using quantized dialect phonemes, syllables, or dialect transcription sequences as weak supervision targets. This eliminates the need for large-scale labeled data and can utilize a small amount of aligned data or pseudo-label data. ,

[0123] in, Input features to the model, The loss is a weakly supervised label sequence; this loss can help the model establish a more stable upper-level temporal output and accelerate model convergence.

[0124] (5-5) Dynamic weight adjustment strategy

[0125] Different training stages have different degrees of dependence on different loss terms. This embodiment adopts the following dynamic weight adjustment strategy:

[0126] Early training phase (1–10 epochs): Increase Reduce the weights to 0.6, decrease other weights, and allow the model to first establish a basic dialect feature recovery capability;

[0127] Mid-training (11–30 epochs): Adjust weights to default values. Balance the various loss constraints;

[0128] Later in training (after 31 epochs): Increase Up to 0.3, increase Reduce to 0.2 Reduce to 0.4 The value was reduced to 0.1, improving the stability and class separability of the representation.

[0129] The above weight values ​​are only the preferred configuration for this embodiment. In actual applications, they can be adjusted according to the linguistic complexity of the target dialect and the scale of the training data.

[0130] The training of this invention does not rely on additional annotations, but only requires pre-statistical analysis of tone sandhi / weakening differences (which can be obtained from a small amount of linguistic knowledge or unlabeled data). Linguistic rules are incorporated into the loss function in the form of soft constraints, rather than hard-coded rules, maintaining the flexibility of end-to-end training of the model. It is particularly suitable for tonal languages, such as Wu, Cantonese, and Min, as well as dialects with complex phonological changes.

[0131] The labeled corpus is obtained by preprocessing the corpus of the target dialect. The preprocessing includes noise removal, uniform sampling rate, and text annotation. Noise removal includes filtering to remove environmental noise, music, reverberation, and other interference.

[0132] The frozen pre-trained model has some of its underlying parameters as parameter layers that retain the learning results of dialect features.

[0133] In this invention, freezing parameters refers to setting the parameters of a portion of the network layers near the input end of the pre-trained model to be non-updateable during the fine-tuning process. These low-level parameters have already learned the common acoustic features of the dialect (such as the mapping from Fbank to tone and phoneme) during the unsupervised pre-training stage. Freezing them can prevent catastrophic forgetting during fine-tuning, thereby preserving the dialect feature learning results. The high-level parameters that are not frozen (several layers near the output end) are updated normally during the fine-tuning process to adapt to the specific labeled corpus.

[0134] This invention also relates to a dialect speech recognition system, comprising:

[0135] The feature extraction module is used to extract multidimensional acoustic features from the speech waveform of the target dialect and construct an acoustic feature sequence.

[0136] The unsupervised pre-training module is used to learn the inherent linguistic structure and feature associations of the target dialect by masking the contents of a partial acoustic feature sequence and using contextual associations to predict the masked part, thereby obtaining a pre-trained model.

[0137] The adjustment module is used to graft a general language model onto the pre-trained model, freeze some of the underlying parameters of the pre-trained model, and adaptively adjust the general language model using labeled corpora to obtain a dialect speech recognition model.

[0138] The present invention also relates to a dialect speech recognition handheld terminal, comprising:

[0139] The acquisition unit is used to acquire dialect speech from users;

[0140] Display unit; and

[0141] The controller, coupled to the acquisition unit and the display unit, is used to perform the dialect speech recognition method on the dialect speech, or to input the dialect speech into the dialect speech recognition system to obtain the corresponding text recognition result; and to control the display unit to display the text recognition result.

[0142] In this invention, the handheld terminal includes, but is not limited to, portable devices with voice acquisition and display functions such as PDAs, smart walkie-talkies, portable translators, smartphones, tablets, and smart voice recorders; the acquisition unit is generally a microphone, and the display unit is generally set in the form of an LCD screen, an OLED screen, or an e-ink screen.

[0143] In this invention, a storage unit and a communication unit are also configured in conjunction with the controller, and the communication unit is associated with a cloud server.

[0144] The following uses the Taizhou dialect as an example to provide a complete implementation case of the present invention.

[0145] Step 1: Data Acquisition and Preprocessing

[0146] We collected folk language data from Jiaojiang, Huangyan, Luqiao, Linhai, Wenling and other districts and counties in Taizhou, including daily dialogues (about 300 hours), folk tales (about 50 hours), and local operas (about 20 hours). All audio was uniformly resampled to 16kHz, mono, and environmental noise was removed using spectral subtraction combined with Wiener filtering.

[0147] Step 2: Feature Extraction

[0148] 80-dimensional Fbank features were extracted using a 25ms frame length and a 10ms frame shift. The fundamental frequency trajectory was obtained based on the RAPT algorithm and normalized to tone values ​​(1–5) to represent the seven tone systems of the Taizhou dialect. Nasalized consonants (such as [ã] and [õ]), as well as special phonemes such as ancient voiced initials and glottal stop codas, were quantitatively extracted. Prosodic features such as logarithmic energy, first-order difference of the fundamental frequency, and duration ratio were extracted for each frame. These features were then concatenated frame by frame into a 128-dimensional feature vector sequence.

[0149] Meanwhile, following the aforementioned method, speech is segmented into syllable-level linguistic units, and seven core linguistic parameters (initials, finals, tones, sound changes, prosody, dialect categories, and six-dimensional continuous features) are extracted to construct a dialect quantitative feature sequence.

[0150] Step 3: Unsupervised pre-training

[0151] Based on the HuBERT Large architecture, it is trained based on the aforementioned masking strategy. Approximately 500 hours of unlabeled Taizhou dialect speech is used, with the masking ratio dynamically adjusted by the global dialect density (50% - 80%), and the unit masking probability dynamically determined by the saliency score (35% - 85%). It is trained for 50 rounds with a learning rate of 1×10 -3 , and a joint loss function (dynamic weight adjustment) is used to obtain the pre-trained model.

[0152] Step 4: Construction of Annotated Corpus

[0153] Select 200 hours from the above collected corpus for professional annotation. Linguistic workers familiar with the Taizhou dialect transcribe the audio into Chinese character text, with the annotation consistency reaching over 95%. Among them, 80% is used for training, 10% for validation, and 10% for testing.

[0154] Step 5: Downstream Fine-tuning

[0155] On the basis of the pre-trained model, graft the Paraformer Large encoder (heads = 8, layers = 50). Map the HuBERT output from 1024 dimensions to 768 dimensions through a linear layer. Freeze the parameters of the bottom 20 - 24 layers of HuBERT, and fine-tune the upper 2 - 4 layers, the linear mapping layer, and the Paraformer model. Use the AdamW optimizer with a learning rate of 1×10 -5 , a batch size of 32, a weight decay of 0.01, and train for 20 rounds. On the test set, the character error rate (CER) drops to 12.5%, significantly better than the baseline model (28.3%) without optimizing using dialect linguistic features.

[0156] Step 6: Deployment on Handheld Terminals

[0157] Quantize and compress the fine-tuned model (model size is approximately 50MB), and deploy it to a handheld PDA device (ARM Cortex-A7, a processor with 4GB RAM and an in-built NPU computing power of 1TOPS). The user presses the voice collection button and says "The weather is nice today" in Taizhou dialect. The device collects the voice (about 2 seconds), and the controller executes the recognition method, and outputs the text "The weather is nice today" within 0.5 seconds and displays it on a 3.5-inch LCD screen. The device supports offline recognition and online model update via Wi-Fi.

[0158] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0160] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0161] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0162] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0163] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A dialect speech recognition method, characterized in that: Multidimensional acoustic features are extracted from the speech waveform of the target dialect, an acoustic feature sequence is constructed, and an unsupervised pre-training method is adopted. By masking part of the acoustic feature sequence and using contextual association to predict the masked part, the inherent linguistic structure and feature association of the target dialect are learned to obtain the pre-trained model. Several target dialects were collected and labeled to obtain labeled corpora. A general language model was grafted onto the pre-trained model, some of the underlying parameters of the pre-trained model were frozen, and the general language model was adaptively adjusted using the labeled corpora to obtain a dialect speech recognition model for dialect speech recognition.

2. The dialect speech recognition method according to claim 1, characterized in that: The multidimensional acoustic features include at least one of the following: basic acoustic features, tone features, special phoneme features, and prosodic features; The multidimensional acoustic features are quantified based on syllable-level linguistic units, and each syllable-level linguistic unit includes at least: The first set of parameters used to characterize the differences in phonetic segments of the target dialect; The second parameter set used to characterize the differences in tone and phonological changes in the target dialect; The third parameter set used to characterize the continuous prosodic features of the target dialect; The third parameter set includes several continuous normalized values ​​that reflect syllable duration, fundamental frequency dynamics, and energy distribution.

3. The dialect speech recognition method according to claim 2, characterized in that: In unsupervised pre-training, masking is performed according to dialect feature modules, which include at least one of tone modules, special phoneme modules, and prosody modules, wherein each dialect feature module corresponds to the acoustic feature to be masked.

4. The dialect speech recognition method according to claim 3, characterized in that: The masking probability in unsupervised pre-training is dynamically determined based on the dialect linguistic saliency of the syllable-level linguistic unit; the dialect linguistic saliency is determined by weighted average of the dialect distinguishing values ​​of the syllable-level linguistic unit at the segment level, tone level, and speech flow prosody level. The masking operation employs multi-level masking granularity, including at least fine-grained masking for single-segment units and prosodic span masking for multiple consecutive syllables; For linguistic units with high presupposition value, reduce the probability of them being obscured.

5. A dialect speech recognition method according to claim 3, characterized in that: The masking content includes continuous segments or discrete points in the acoustic feature sequence; the proportion of the masked part to the total length of the acoustic feature sequence is dynamically adjusted according to the linguistic complexity of the target dialect.

6. The dialect speech recognition method according to claim 1, characterized in that: A joint loss function for unsupervised pre-training is constructed, which includes several sub-loss terms constrained by the linguistic characteristics of the target dialect. The total loss is formed by the weighted sum of the sub-loss terms.

7. A dialect speech recognition method according to claim 1, characterized in that: The labeled corpus is obtained by preprocessing the corpus of the target dialect, including noise removal, uniform sampling rate, and text annotation.

8. The dialect speech recognition method according to claim 1, characterized in that: The frozen pre-trained model has some of its underlying parameters as parameter layers that retain the learning results of dialect features.

9. A dialect speech recognition system, characterized in that: include: The feature extraction module is used to extract multidimensional acoustic features from the speech waveform of the target dialect and construct an acoustic feature sequence. The unsupervised pre-training module is used to learn the inherent linguistic structure and feature associations of the target dialect by masking the contents of a partial acoustic feature sequence and using contextual associations to predict the masked part, thereby obtaining a pre-trained model. The adjustment module is used to graft a general language model onto the pre-trained model, freeze some of the underlying parameters of the pre-trained model, and adaptively adjust the general language model using labeled corpora to obtain a dialect speech recognition model.

10. A dialect speech recognition handheld terminal, characterized in that: include: The acquisition unit is used to acquire dialect speech from users; Display unit; as well as The controller, coupled to the acquisition unit and the display unit, is used to perform the dialect speech recognition method according to any one of claims 1 to 8 on the dialect speech, or to input the dialect speech into the dialect speech recognition system according to claim 9 to obtain the corresponding text recognition result; and to control the display unit to display the text recognition result.

Images