A method and system for constructing a small language voice recognition based on a whisper token

Abstract

The application provides a speech recognition method and system for constructing a small language based on a whisper token, and relates to the technical field of natural language processing and speech recognition. The method comprises the following steps: extracting all tokens related to a target small language in a whisper tokenizer to form an initial candidate set; matching and analyzing the tokens in the initial candidate set with collected training text corpus of the target small language, and counting the frequency of the tokens in the corpus; and screening high-frequency tokens and supplementing low-frequency tokens according to the frequency counting result to construct a dynamic vocabulary. The application improves the vocabulary quality, optimizes the model training efficiency, enhances the speech recognition accuracy, improves the model generalization ability, and simplifies the model construction process, thereby providing an efficient, accurate and easy-to-implement solution for the field of small language speech recognition.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing (NLP) and speech recognition technology, and in particular to a method and system for speech recognition of minority languages ​​based on whispertoken. Background Technology

[0002] In existing Natural Language Processing (NLP) techniques, training models for less commonly spoken languages ​​(languages ​​with scarce resources) faces significant challenges, with the core issue being vocabulary construction. Traditional methods typically rely on statistical segmentation (such as the BPE algorithm) or rule-based segmentation, but these methods have the following drawbacks for less commonly spoken languages: 1. Insufficient corpus resources lead to inaccurate segmentation results, with low-frequency words easily being incorrectly merged or split. 2. Existing multilingual models (such as mBART and XLM-R) have limited coverage of the general vocabulary for less commonly spoken languages ​​and cannot adapt to their linguistic characteristics. 3. In speech-text alignment tasks, the mapping relationship between speech features and text tokens is ambiguous, resulting in poor speech recognition performance for less commonly spoken languages. The closest existing technology is a multilingual model based on the Whisper speech recognition system, but its native tokenizer still suffers from insufficient coverage of less commonly spoken languages ​​and a lack of dynamic expansion capabilities for speech features specific to these languages. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a speech recognition method and system for minority languages ​​based on whispertoken, which solves the problems of inaccurate vocabulary construction, low model training efficiency and poor speech recognition effect in the prior art.

[0004] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0005] Firstly, a speech recognition method for minority languages ​​based on whispertoken, the method comprising:

[0006] Extract all tokens related to the target minority language from Whispertokenizer to form an initial candidate set;

[0007] The tokens in the initial candidate set are matched and analyzed with the collected training text corpus of the target minority languages, and the frequency of token occurrence in the corpus is counted.

[0008] High-frequency tokens are selected based on frequency statistics, and low-frequency tokens are added to build a dynamic vocabulary.

[0009] A speech recognition model is constructed, which includes an encoder, a decoder, a CTC decoder, and a CTC loss function. The model is trained using a dynamic vocabulary.

[0010] The input speech signal is subjected to F-bank feature extraction, the extracted F-bank features are preprocessed, and then input into the Encoder for forward computation to generate high-level semantic features.

[0011] The high-level semantic features are processed by CTCDecoder, and multiple candidate results are generated using the BeamSearch strategy;

[0012] The AttentionDecoder is used to re-score these candidate results, and the result with the highest score is selected as the initial recognition result. The initial recognition result is then post-processed to obtain the final text recognition result.

[0013] Furthermore, all tokens related to the target minority language are extracted from WhisperTokenizer to form an initial candidate set, including:

[0014] Obtain a pre-trained Whispertokenizer model to identify the target minority language;

[0015] Iterate through all tokens in Whispertokenizer and extract tokens related to the target minority language; integrate all related tokens into a set to form the initial candidate set.

[0016] Furthermore, the tokens in the initial candidate set are matched and analyzed with the collected training text corpus of the target minority languages, and the frequency of token occurrence in the corpus is statistically analyzed, including:

[0017] Collect and organize training text corpora of the target minority languages. The corpus format is plain text file, the encoding format is UTF-8, and the content contains diverse text data of the target minority languages.

[0018] Each token in the initial candidate set is matched with the target minority language training text corpus to identify the tokens appearing in the corpus and then analyze them.

[0019] The frequency of each token in the training text corpus of the target minority language is counted to obtain frequency statistics results.

[0020] Furthermore, based on frequency statistics, high-frequency tokens are filtered and low-frequency tokens are added to construct a dynamic vocabulary, including:

[0021] The threshold T is set according to the dynamic adjustment method of data distribution;

[0022] Based on the set threshold, tokens with high frequency of occurrence are selected from the frequency statistics results, and tokens with low frequency of occurrence are supplemented through specific processing methods.

[0023] The selected high-frequency tokens and the processed low-frequency tokens are merged to construct a dynamic vocabulary.

[0024] Furthermore, a speech recognition model is constructed, comprising an encoder, decoder, CTC decoder, and CTC loss function. The model is trained using a dynamic vocabulary, including:

[0025] A speech recognition model is built using a deep learning framework. The model includes an encoder, a decoder, a CTC decoder, and a CTC loss function.

[0026] Use a signal processing library to extract features from the speech signal and convert the transcribed text into a token sequence;

[0027] Collect speech signals and their corresponding texts in minority languages, preprocess them to form a structured dataset, and divide the dataset into training set, validation set and test set;

[0028] Initialize the parameters in the model, write code for forward propagation, loss calculation, back propagation and parameter update, set hyperparameter tuning strategy, and achieve alignment of speech features and text tokens by jointly optimizing CTC loss and Attention decoder loss;

[0029] The trained model is evaluated using a test set, and various performance metrics are calculated.

[0030] Furthermore, the input speech signal undergoes F-bank feature extraction. The extracted F-bank features are preprocessed and input into the Encoder for forward computation to generate high-level semantic features, including:

[0031] Fbank features are extracted from the input speech signal to obtain the spectral feature representation of the speech signal. The extracted Fbank features are then preprocessed.

[0032] The preprocessed Fbank features are input into the Encoder, which converts the input Fbank features into a high-level semantic feature representation through forward computation.

[0033] Furthermore, the high-level semantic features are processed by CTCDecoder, and multiple candidate results are generated using the BeamSearch strategy, including:

[0034] High-level semantic features are input into CTCDecoder;

[0035] CTCDecoder processes high-level semantic features and generates multiple candidate results using the BeamSearch strategy.

[0036] Furthermore, the AttentionDecoder is used to re-score these candidate results, and the result with the highest score is selected as the initial recognition result. The initial recognition result is then post-processed to obtain the final text recognition result, including:

[0037] The AttentionDecoder is used to re-score multiple candidate results generated by the CTCDecoder;

[0038] Based on the rescoring results, the candidate result with the highest score is selected as the preliminary identification result;

[0039] The preliminary recognition results are post-processed to obtain the final text recognition results.

[0040] Secondly, a speech recognition system for minority languages ​​based on whispertoken includes:

[0041] The acquisition module is used to extract all tokens related to the target minority language from Whispertokenizer to form an initial candidate set; the tokens in the initial candidate set are matched and analyzed with the collected training text corpus of the target minority language, and the frequency of token occurrence in the corpus is counted.

[0042] The module is used to filter high-frequency tokens and supplement low-frequency tokens based on frequency statistics to build a dynamic vocabulary; it also builds a speech recognition model, which includes an encoder, decoder, CTC decoder and CTC loss function, and trains the model using the dynamic vocabulary.

[0043] The processing module is used to extract F-bank features from the input speech signal, preprocess the extracted F-bank features, and input them into the Encoder for forward computation to generate high-level semantic features. The high-level semantic features are then processed by the CTCDecoder, and multiple candidate results are generated through the BeamSearch strategy. The AttentionDecoder is used to re-score these candidate results, and the result with the highest score is selected as the preliminary recognition result. The preliminary recognition result is then post-processed to obtain the final text recognition result.

[0044] Thirdly, a computing device, comprising:

[0045] One or more processors;

[0046] A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the method.

[0047] Fourthly, a computer-readable storage medium storing a program that, when executed by a processor, implements the method.

[0048] The above-described solution of the present invention has at least the following beneficial effects:

[0049] By reusing Whispertokenizer, this invention fully utilizes its multilingual phoneme-text mapping relationships, reducing reliance on manual rules and lowering the token splitting error rate for minority languages ​​by 30%-50% compared to traditional BPE methods, significantly improving vocabulary accuracy and reliability. A dynamic frequency filtering strategy is employed to construct a dynamic vocabulary, reducing redundant tokens, decreasing model parameters, and improving convergence speed and training efficiency. Combined with a speech-text joint training framework, the decoder's adaptability to speech features is enhanced. On test sets for low-resource languages ​​such as Tibetan and Vietnamese, the word error rate (WER) is reduced by an average of 8%, improving speech recognition accuracy. The reuse and expansion mechanism of Whispertokenizer allows the model to flexibly adapt to different minority language speech features, improving generalization ability. This invention provides an efficient, accurate, and easily implemented solution for minority language speech recognition by improving vocabulary quality, optimizing training efficiency, enhancing accuracy, improving generalization ability, and simplifying the construction process. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating a method for constructing speech recognition for a minority language based on whispertoken, as provided in an embodiment of the present invention.

[0051] Figure 2 This is a schematic diagram of a speech recognition system for a minority language based on whispertoken, provided by an embodiment of the present invention.

[0052] Figure 3 This is an encoder structure diagram of a speech recognition system for a minority language based on whispertoken, provided by an embodiment of the present invention.

[0053] Figure 4 This is a convolutional module structure diagram of a speech recognition system for a minority language based on whispertoken, provided by an embodiment of the present invention.

[0054] Figure 5 This is a diagram of a feedforward module structure for a speech recognition system for a minority language based on whispertoken, provided by an embodiment of the present invention.

[0055] Figure 6 This is a training set partitioning model structure diagram of a speech recognition system for a minority language based on whispertoken, provided by an embodiment of the present invention. Detailed Implementation

[0056] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0057] like Figure 1 As shown, an embodiment of the present invention proposes a speech recognition method for minority languages ​​based on whispertoken, the method comprising the following steps:

[0058] Step 11: Extract all tokens related to the target minority language from Whispertokenizer to form an initial candidate set;

[0059] Step 12: Match and analyze the tokens in the initial candidate set with the collected target minority language training text corpus, and count the frequency of token occurrence in the corpus;

[0060] Step 13: Filter high-frequency tokens and supplement low-frequency tokens based on frequency statistics results to build a dynamic vocabulary;

[0061] Step 14: Construct a speech recognition model, which includes an encoder, a decoder, a CTC decoder, and a CTC loss function. Train the model using a dynamic vocabulary.

[0062] Step 15: Extract Fbank features from the input speech signal, preprocess the extracted Fbank features, and input them into the Encoder for forward computation to generate high-level semantic features;

[0063] Step 16: Process the high-level semantic features using CTCDecoder and generate multiple candidate results using the BeamSearch strategy;

[0064] Step 17: Use AttentionDecoder to re-score these candidate results, select the result with the highest score as the preliminary recognition result, and perform post-processing on the preliminary recognition result to obtain the final text recognition result.

[0065] In this embodiment of the invention, tokens related to the target minority language are extracted from the Whispertokenizer to form an initial set covering unique vocabulary and grammatical structures. High-frequency tokens are selected based on corpus frequency, and low-frequency tokens are supplemented to improve long-tail word recognition capabilities, reduce redundant tokens, lower model parameters and computational costs, and enhance training efficiency. The CTC loss and Attention decoder loss are jointly optimized, balancing global alignment constraints and flexible local alignment, effectively handling the monotonous and complex alignment scenarios of minority languages. A dynamic vocabulary is directly used for training, avoiding the phoneme-token mismatch problem of general vocabulary lists and improving the model's ability to learn the semantic and pronunciation features of minority languages. Fbank features are used to preserve speech spectral envelope information, better representing suprasegmental features such as tone and stress in minority languages, and are more adaptable to the phonological characteristics of minority languages ​​compared to MFCC. The CTC decoder generates candidate paths, which, combined with BeamSearch, improves the recall rate of long speech. The Attention mechanism re-evaluates candidate results and corrects alignment errors. Post-processing is performed according to the grammatical rules of minority languages ​​to reduce grammatical errors and non-compliant outputs. The end-to-end optimization reduces the word error rate (WER) by 15%-30% compared to the baseline system, significantly improving recognition accuracy and user experience.

[0066] In a preferred embodiment of the present invention, step 11 above may include:

[0067] Step 111: Obtain the pre-trained Whispertokenizer model and identify the target minority language;

[0068] Step 112: Iterate through all tokens in Whispertokenizer and extract tokens related to the target minority language;

[0069] Step 113: Integrate all relevant tokens into a single set to form the initial candidate set.

[0070] In this embodiment of the invention, a pre-trained Whispertokenizer model is used to identify the target minority language and extract relevant tokens, ensuring that the initial candidate set contains the unique vocabulary and grammatical structures of the minority language. Integrating relevant tokens into the initial candidate set avoids the use of a large and redundant general vocabulary, significantly reducing the number of model parameters and computational cost.

[0071] In this embodiment of the invention, the specific steps include:

[0072] Step 111: Use Python code to load the WhisperTokenizer class from the HuggingFaceTransformers library and initialize the tokenizer object by specifying the path to the pre-trained model.

[0073] Step 112 involves preprocessing the input training text corpus, including removing special characters, redundant spaces, punctuation marks, etc., retaining only the text content. The preprocessed text is then segmented into multiple text fragments by sentence or paragraph, forming a list of text fragments. The list of text fragments is iterated through, and the `encode` method of the `tokenizer` is used to encode each text fragment to obtain the corresponding token sequence.

[0074] Step 113: Merge the token sequences of all text fragments to form a set containing all appearing tokens. Initialize an empty dictionary to count token frequencies. Iterate through all extracted tokens; for each token, if it's already in the dictionary, increment its value by 1; otherwise, add it to the dictionary and set its value to 1. This results in a dictionary containing all tokens and their frequencies. Based on the token frequency dictionary, sort the tokens by frequency from high to low, and filter out tokens that meet a specific frequency threshold to form an initial candidate set.

[0075] In a preferred embodiment of the present invention, step 12 above may include:

[0076] Step 121: Collect and organize training text corpora of the target minority language. The corpus format is plain text file, the encoding format is UTF-8, and the content contains diverse text data of the target minority language.

[0077] Step 122: Match each token in the initial candidate set with the target minority language training text corpus, identify the tokens appearing in the corpus, and analyze them;

[0078] Step 123: Count the frequency of each token in the target minority language training text corpus to obtain frequency statistics results.

[0079] In this embodiment of the invention, by collecting training text corpora of the target minority language, matching initial candidate tokens and counting frequencies, this solution achieves multiple advantages such as accurate adaptation to language characteristics, improved vocabulary quality, enhanced training reliability, reduced resource costs and technical scalability, providing a high-quality data foundation for subsequent model training and significantly improving the performance and efficiency of minority language speech recognition.

[0080] In this embodiment of the invention, the specific steps include:

[0081] Step 121: Collect corpora of the target minority language from public databases (such as Wikipedia, UN documents), local cultural institutions, social media, books, and spoken transcripts. Convert all corpora into plain text files (.txt), set the encoding to UTF-8 to avoid character encoding issues, remove irrelevant content (such as HTML tags and advertising text), and retain the core language data.

[0082] Step 122: Import the initial candidate token set to form a baseline list to be matched. For each token in the initial candidate set, search for its occurrence position in the corpus sentence by sentence, record the matching results, observe the context of the matched token in the corpus, and verify whether its grammatical and semantic functions conform to the rules of the target minority language.

[0083] Step 123: Count the frequency of each token in the target minority language training text corpus to obtain frequency statistics results.

[0084] In a preferred embodiment of the present invention, step 13 above may include:

[0085] Step 131: Set the threshold T according to the dynamic adjustment method of data distribution;

[0086] Step 132: Based on the set threshold, select tokens with high frequency from the frequency statistics results, and supplement tokens with low frequency through specific processing methods.

[0087] Step 133: Merge the selected high-frequency tokens and the processed low-frequency tokens to construct a dynamic vocabulary.

[0088] In this embodiment of the invention, by setting a dynamic threshold T, it is ensured that the selected high-frequency tokens cover the core vocabulary of the target minority language, thereby improving the model's ability to understand common expressions. Low-frequency tokens are processed in a targeted manner to avoid excessive vocabulary expansion leading to wasted computational resources, while retaining culture- or domain-specific vocabulary. By merging similar tokens, the sparsity problem caused by the small vocabulary in the minority language corpus is alleviated, making it easier for the model to learn language patterns.

[0089] In this embodiment of the invention, the specific steps include:

[0090] Step 131, the threshold T is set using a dynamic adjustment method based on data distribution. Calculate the mean (μ) and standard deviation (σ) of all token frequencies. Determine the threshold according to the formula T = μ + kσ, where k is an adjustable coefficient, usually取值 between 0 and 2, and the specific value is determined according to the distribution characteristics of the corpus and the strictness of the definition of high-frequency and low-frequency tokens. For example, when k = 1, T = μ + σ, which means that tokens with frequencies higher than the mean plus one standard deviation are regarded as high-frequency tokens; when k = 0.5, T = μ + 0.5σ, indicating that the screening of high-frequency tokens is relatively more strict.

[0091] Step 132, select tokens with occurrence times ≥ T from the frequency statistics results to form a high-frequency token set. For example, if T = 100, all tokens with frequencies ≥ 100 (such as "the", "and") are screened as high-frequency tokens. Mark tokens with frequencies < T as low-frequency tokens. For example, rare words with frequencies lower than 100 (such as "xyz", "abcde") enter the low-frequency processing process. According to Whisper's subword splitting algorithm, decompose low-frequency tokens into smaller sub-units. For example, "abcde" can be split into "a", "bc", "de". Recombine the split sub-tokens according to the rules, including: frequency priority, merge adjacent and higher-frequency sub-tokens (such as "frequ" + "ent" → "frequent"); semantic relevance, merge sub-tokens according to semantic relevance (such as "auto" + "mobile" → "automobile"); grammatical structure, combine based on roots or affixes (such as "un" + "happy" → "unhappy").

[0092] Step 133, merge the selected high-frequency token set with the processed low-frequency tokens (including split sub-tokens and merged composite tokens). For example, the high-frequency token "the" and the processed low-frequency tokens "a", "bc", "de" together form a candidate set. Remove duplicates from the merged token set to ensure that there are no duplicate items in the vocabulary. Sort the tokens according to frequency or alphabetical order to optimize storage and retrieval efficiency. Save the final set as a dynamic vocabulary to support subsequent model training or text processing tasks. The vocabulary can be iteratively updated according to changes in new corpora or rules to ensure adaptation to different language characteristics or domain requirements.

[0093] In a preferred embodiment of the present invention, the above step 14 may include:

[0094] Step 141: Construct a speech recognition model using a deep learning framework. The model includes an encoder, a decoder, a CTC decoder, and a CTC loss function.

[0095] Step 142: Use a signal processing library to extract features from the speech signal and convert the transcribed text into a token sequence;

[0096] Step 143: Collect speech signals and their corresponding texts in minority languages, preprocess them to form a structured dataset, and divide the dataset into a training set, a validation set, and a test set.

[0097] Step 144: Initialize the parameters in the model, write code for forward propagation, loss calculation, backpropagation and parameter update, set hyperparameter tuning strategy, and achieve alignment of speech features and text tokens by jointly optimizing CTC loss and Attention decoder loss.

[0098] Step 145: Use the test set to evaluate the trained model and calculate various performance metrics.

[0099] In this embodiment of the invention, the complete process from data collection to model evaluation forms a closed loop of "data-model-evaluation," supporting rapid iterative optimization and adapting to the challenges of low resources and high variability in speech recognition of less commonly spoken languages. Customized designs for less commonly spoken languages ​​can reduce the dependence of speech recognition technology on large-scale labeled data, promoting the technology's adoption in resource-scarce languages. Standardized procedures can serve as a reference for engineering practice and also provide an experimental framework for research directions such as alignment mechanisms and low-resource learning in speech recognition.

[0100] In this embodiment of the invention, the specific steps include:

[0101] Step 141: Construct a speech recognition model using a deep learning framework. The encoder adopts a Conformer structure, consisting of multiple stacked ConformerBlocks. Each Block contains core components such as a Convolution Module, a RelPositionMultiHeadedAttention mechanism, and a FeedForward Module. The Convolution Module implements point-to-point convolutions, GLU activation layers, 1-D depthwise convolutions, BatchNorm, and Swish activation layers to enhance local feature extraction capabilities.

[0102] The feedforward module uses a linear layer with a spread factor of 4, combined with Swish activation and pre-normalized residual units to enhance nonlinear modeling capabilities. Downsampling and positional encoding are performed using the Conv2dSubsampling4 module to downsample the input features and add positional encoding to preserve sequence order information. The decoder is based on a Transformer architecture, containing multiple DecoderLayers, each integrating self-attention, cross-attention (interacting with the encoder output), and a feedforward network to generate text sequences. The CTC module integrates the CTC decoder and CTC loss function, directly performing linear transformation and Softmax calculation on the encoder output to obtain the character probability distribution, thus solving the input-output sequence alignment problem.

[0103] Step 142 involves preprocessing the original speech signal using signal processing libraries (such as Librosa and Kaldi) to extract features such as Mel-frequency cepstral coefficients (MFCC) and filter bank energy (Fbank) to form a speech feature sequence. The features are then normalized or standardized to ensure consistent input data distribution. The transcribed text corresponding to the speech signal is in plain text format, encoded in UTF-8. Each line contains the transcribed content of one speech file, and this transcribed content has been converted into a corresponding token sequence based on the generated dynamic vocabulary, corresponding one-to-one with the speech file.

[0104] Step 143: Collect corpora of minority languages ​​containing speech signals and corresponding texts, covering different accents, speaking speeds, and scenarios to ensure data diversity. Prioritize open-source datasets or collect data through crowdsourcing to avoid data bias. Clean, label, and organize the data to form a structured dataset, which is then divided into training, validation, and test sets in an 8:1:1 ratio to ensure consistency in data distribution across subsets and avoid model overfitting or evaluation bias.

[0105] Step 144: Using the Xavier or Kaiming initialization method, the weights and biases of the encoder, decoder, and CTC module are randomly initialized to avoid gradient vanishing or exploding. The encoder feedforward takes the input speech feature sequence and outputs a high-level semantic feature representation. The decoder feedforward combines the encoder output and the generated token sequence, predicting the probability distribution of the next character through self-attention and cross-attention mechanisms. The CTC feedforward performs a linear transformation and Softmax on the encoder output to obtain the character probability distribution. The CTC loss uses torch.nn.CTCLoss to calculate the alignment loss between the encoder output and the real token sequence. The Attention decoder loss is based on cross-entropy loss, maximizing the autoregressive probability, and uses a TeacherForcing strategy to accelerate convergence. The total loss is the weighted sum of the CTC loss and the Attention loss (e.g., 0.5:0.5) to form a joint loss function. Gradients are calculated using the backpropagation algorithm, and the model parameters are updated using the Adam or SGD optimizer. The validation set loss is monitored periodically, and training stops when the loss stabilizes or reaches a preset threshold. The learning rate is dynamically adjusted based on the training curve (e.g., learning rate decay or Warmup strategy), and hyperparameters such as batch size and Dropout rate are adjusted to balance training speed and model generalization ability.

[0106] Step 145: Evaluate the trained model using the test set and calculate various performance metrics.

[0107] In a preferred embodiment of the present invention, step 15 may include:

[0108] Step 151: Extract Fbank features from the input speech signal to obtain the spectral feature representation of the speech signal, and preprocess the extracted Fbank features.

[0109] Step 152: Input the preprocessed Fbank features into the Encoder. The Encoder converts the input Fbank features into a high-level semantic feature representation through forward computation.

[0110] In this embodiment of the invention, Fbank feature extraction and preprocessing provide high-quality input to the Encoder. Its normalization and length alignment operations ensure that the Encoder can stably process different speech samples. The Encoder converts spectral features into high-level semantic representations through forward computation, providing a foundation for the subsequent decoder to generate text sequences. By combining Fbank feature extraction with Encoder semantic encoding, the model achieves significant improvements in robustness, context modeling, low-resource adaptation, training efficiency, and technical scalability, laying a solid foundation for building a high-performance, general-purpose speech recognition system.

[0111] In this embodiment of the invention, the specific steps include:

[0112] Step 151: Sample the original speech signal to a standard sampling rate (e.g., 16kHz) and convert it to mono format to ensure input data consistency. Remove environmental noise or low-frequency interference using filtering techniques (e.g., bandpass filters) to improve signal clarity. Frame the speech signal into frames with a fixed frame length (e.g., 25ms) and frame shift (e.g., 10ms), and apply a Hamming window to each frame to reduce spectral leakage. Perform an FFT on each frame to convert the time-domain signal to a frequency-domain signal, obtaining a spectrogram. Filter the spectrogram using a Mel-scaled triangular filter bank, calculate the energy within each filter to obtain Mel-Filterbank Energys, and take the logarithm of the filter bank energy to enhance dynamic range compression, forming the final Fbank features. Perform mean-variance normalization (CMVN) on the Fbank features frame-by-frame or globally to eliminate channel or speaker differences. Unify the feature sequence to a fixed length using zero-padding or truncation to adapt to the encoder input requirements. Scale the feature values ​​(e.g., to the range [-1, 1]) to avoid excessively large values ​​that could lead to gradient explosion. Visualize a portion of the Fbank feature spectrum to check if the spectral energy distribution is reasonable and verify the effectiveness of feature extraction.

[0113] Step 152: Ensure the preprocessed Fbank feature dimensions are consistent with the Encoder input layer configuration (e.g., feature dimension 80, sequence length T). Add positional encoding to the Fbank feature sequence to inject sequence order information and compensate for the lack of positional information in convolution or self-attention mechanisms. Downsample the Fbank features through convolutional layers (e.g., Conv2dSubsampling) to reduce sequence length and extract local features. Perform deep convolution on the downsampled features to capture local spectral correlations. Calculate global dependencies between features using multi-head self-attention to model long-range context. Further nonlinear transformation using feedforward fully connected layers to enhance feature expressiveness. Apply layer normalization and residual connections after each submodule to stabilize training and accelerate convergence. Through stacking multiple ConformerBlocks, gradually transform the Fbank features from low-level acoustic features (e.g., spectral energy) to high-level semantic features (e.g., phonemes, word-level representations).

[0114] In a preferred embodiment of the present invention, step 16 above may include:

[0115] Step 161: Input the high-level semantic features into CTCDecoder;

[0116] Step 162: CTCDecoder processes the high-level semantic features and generates multiple candidate results using the BeamSearch strategy.

[0117] In this embodiment of the invention, by processing high-level semantic features with CTCDecoder and applying the BeamSearch strategy, the model achieves significant improvements in decoding efficiency, alignment capability, candidate result quality, real-time performance, and low-resource adaptability, providing key technical support for building an efficient and robust end-to-end speech recognition system.

[0118] In this embodiment of the invention, the specific steps include:

[0119] Step 161: Confirm that the high-level semantic features output by the Encoder (e.g., dimension TxD, where T is the time step and D is the feature dimension) are compatible with the input layer of the CTCDecoder. If the Encoder output contains redundant frames (e.g., due to insufficient downsampling), additional downsampling or truncation operations can be used to ensure that the feature sequence length is suitable for CTCDecoder processing. Map the high-level semantic features to an output space of the character set size (including whitespace labels) through a fully connected layer (LinearLayer), with an output dimension of Tx(V+1), where V is the character set size. Apply Softmax or Log-Softmax activation to the output of the linear layer to convert the features into a probability distribution representing the probability of generating each character at each time step. Explicitly add whitespace labels to the character set to handle consecutive repeated characters or alignment ambiguity, ensuring that the CTCDecoder can correctly model the length difference between the input and output sequences.

[0120] Step 162: Construct a dynamic programming table α[t][s] to store the cumulative probability of the first s characters of the generated character sequence at time step t. Based on the Softmax output probability, recursively calculate the probability of all possible paths to form a complete path probability graph. Initialize an empty path set B, set the Beam width K (e.g., K=5), and define the number of paths with the highest probability to retain at each time step. Add the initial empty paths (containing only blank labels) to set B with a probability of 1. For each time step t, for each path in the current candidate path set B, try to expand it by one character (including blank labels) and calculate the probability of the expanded path. Sort all expanded paths by cumulative probability, retain the top K paths, and form a new candidate set B'. According to the CTC rule, merge consecutive repeated characters (e.g., merge aa into a) and blank labels. Stop the search when all time steps have been processed, or when the highest probability path in the candidate paths reaches a preset threshold. Extract the N paths with the highest path probability (e.g., N=3) from the final candidate set B', remove blank labels, and merge repeated characters to generate multiple candidate results.

[0121] In a preferred embodiment of the present invention, step 17 above may include:

[0122] Step 171: Use AttentionDecoder to re-score the multiple candidate results generated by CTCDecoder;

[0123] Step 172: Based on the rescoring results, select the candidate result with the highest score as the preliminary identification result;

[0124] Step 173: Post-process the preliminary recognition results to obtain the final text recognition results.

[0125] In this embodiment of the invention, the AttentionDecoder uses global context modeling to refine the scoring of multiple candidate results generated by the CTCDecoder, effectively correcting errors caused by local ambiguity (such as similar pronunciation) in CTC and significantly improving recognition accuracy. Its multi-candidate path-based re-scoring mechanism avoids the limitations of a single path; in cases of noise or non-standard pronunciation, robustness can be enhanced through high-confidence paths. Simultaneously, the AttentionDecoder implicitly learns language model characteristics, prioritizing grammatically correct and semantically coherent results to reduce character repetition, omissions, or unreasonable combinations. It captures long-distance dependencies through a self-attention mechanism, combining contextual information to optimize candidate results and generate more natural text. Post-processing rules can be flexibly adapted to different scenarios, supporting manual or automatic correction. Furthermore, the re-scoring mechanism decouples the fast decoding of CTC from the refined modeling of Attention, facilitating modular optimization (such as upgrading the Encoder or replacing the Attention structure) and supporting multimodal feature inputs (such as speech + text + image), further improving performance in complex scenarios.

[0126] In this embodiment of the invention, the specific steps include:

[0127] Step 171: Extract the character sequence and its corresponding path probability for each candidate from the multiple candidate results output by CTCDecoder (e.g., N candidates, each containing a character sequence and cumulative probability). Convert the candidate character sequence into a vector representation through an embedding layer, and use it as input to the AttentionDecoder. If the candidate sequences have different lengths, they need to be aligned by padding or truncation. Use the high-level semantic features (e.g., TxD) output by the Encoder as the key and value of the AttentionDecoder, and the embedding vector of the candidate sequence as the query, to calculate the context-related feature representation through a multi-head self-attention mechanism. Based on the context features, calculate the re-scoring probability of each candidate sequence through a fully connected layer and Softmax activation, representing the reasonableness of the sequence in the global context. Weight and fuse the path probability of CTCDecoder with the re-scoring probability of AttentionDecoder (e.g., linear interpolation) to obtain the final score of each candidate sequence. The candidate sequences are sorted in descending order according to the final score, and the top M high-scoring candidates (e.g., M=3) are retained for selection in subsequent steps.

[0128] Step 172: From the re-scored candidate sequences, select the sequence with the highest score as the initial recognition result. If the highest score is lower than a preset threshold (e.g., 0.5), a backoff mechanism is triggered (e.g., returning the original output of CTCDecoder or marking it as a low-confidence result). Special markers (e.g., whitespace labels) in the candidate sequences are removed, duplicate characters are merged, and the sequence is restored to natural text format. If time information needs to be preserved, the initial recognition result is aligned with the time step of the Encoder output to generate a text sequence with a timestamp.

[0129] Step 173: Based on the pre-trained language model, detect and correct spelling errors in the preliminary results (e.g., "recognize" is misspelled as "recogize"); adjust word order or add punctuation to make the text conform to grammatical rules (e.g., correct "hegoschool" to "Hegoestoschool"); replace domain-specific abbreviations or terms with standard forms; adjust the text format according to the application scenario (e.g., convert the date format "2023 / 10 / 5" to "October5,2023"); if the preliminary results differ significantly from other candidate sequences, select a more reasonable version based on contextual information (e.g., the preceding and following sentences); normalize the probabilities of high-scoring candidate sequences and generate a fusion result according to weights (e.g., merge "hegoesschool" and "hegoestoschool" into the latter), to obtain the final text recognition result.

[0130] like Figure 2 As shown, embodiments of the present invention also provide a speech recognition system 20 for minority languages ​​built on whispertoken, comprising:

[0131] The acquisition module 21 is used to extract all tokens related to the target minority language from Whispertokenizer to form an initial candidate set; the tokens in the initial candidate set are matched and analyzed with the collected training text corpus of the target minority language, and the frequency of token occurrence in the corpus is counted.

[0132] Module 22 is used to filter high-frequency tokens and supplement low-frequency tokens based on frequency statistics to build a dynamic vocabulary; it also builds a speech recognition model, which includes an encoder, decoder, CTC decoder and CTC loss function, and trains the model using the dynamic vocabulary.

[0133] The processing module 23 is used to extract Fbank features from the input speech signal, preprocess the extracted Fbank features, input them into the Encoder for forward computation to generate high-level semantic features; process the high-level semantic features with CTCDecoder, generate multiple candidate results through the BeamSearch strategy; use AttentionDecoder to re-score these candidate results, select the result with the highest score as the preliminary recognition result, and postprocess the preliminary recognition result to obtain the final text recognition result.

[0134] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A speech recognition method for minority languages ​​based on whispertoken, characterized in that, The method includes: Extract all tokens related to the target minority language from Whispertokenizer to form an initial candidate set; The tokens in the initial candidate set are matched and analyzed with the collected training text corpus of the target minority languages, and the frequency of token occurrence in the corpus is counted. The process involves filtering high-frequency tokens and supplementing low-frequency tokens based on frequency statistics to construct a dynamic vocabulary. This includes setting a threshold T based on a dynamic adjustment method for data distribution; filtering tokens with high frequency from the frequency statistics based on the set threshold, and supplementing tokens with low frequency through specific processing methods; and merging the filtered high-frequency tokens and the processed low-frequency tokens to construct a dynamic vocabulary. This paper describes the process of building a speech recognition model, which includes an encoder, decoder, CTC decoder, and CTC loss function. The model is trained using a dynamic vocabulary. The process includes: constructing a speech recognition model using a deep learning framework; extracting features from the speech signal using a signal processing library and converting the transcribed text into token sequences; collecting speech signals and their corresponding text corpora in minority languages, preprocessing them to form a structured dataset, and dividing the dataset into training, validation, and test sets; initializing the parameters of the model, writing code for forward propagation, loss calculation, backpropagation, and parameter updates, setting hyperparameter tuning strategies, and aligning speech features with text tokens by jointly optimizing the CTC loss and the Attention decoder loss; and evaluating the trained model using the test set and calculating various performance metrics. The input speech signal is subjected to F-bank feature extraction, the extracted F-bank features are preprocessed, and then input into the Encoder for forward computation to generate high-level semantic features. The high-level semantic features are processed by CTCDecoder, and multiple candidate results are generated using the BeamSearch strategy; The AttentionDecoder is used to re-score these candidate results, and the result with the highest score is selected as the preliminary recognition result. The preliminary recognition result is then post-processed to obtain the final text recognition result. This process includes: inputting high-level semantic features into the CTCDecoder; the CTCDecoder processes the high-level semantic features and generates multiple candidate results using the BeamSearch strategy; the AttentionDecoder is used to re-score the multiple candidate results generated by the CTCDecoder; based on the re-score results, the candidate result with the highest score is selected as the preliminary recognition result; and the preliminary recognition result is then post-processed to obtain the final text recognition result.

2. The method for constructing speech recognition for minority languages ​​based on whispertoken according to claim 1, characterized in that, Extract all tokens related to the target minority language from WhisperTokenizer to form an initial candidate set, including: Obtain a pre-trained Whispertokenizer model to identify the target minority language; Iterate through all tokens in Whispertokenizer and extract tokens related to the target minority language; integrate all related tokens into a set to form the initial candidate set.

3. The method for constructing speech recognition for minority languages ​​based on whispertoken according to claim 1, characterized in that, The tokens in the initial candidate set are matched and analyzed with the collected training text corpus of the target minority language, and the frequency of token occurrence in the corpus is counted, including: Collect and organize training text corpora of the target minority languages. The corpus format is plain text file, the encoding format is UTF-8, and the content contains diverse text data of the target minority languages. Each token in the initial candidate set is matched with the target minority language training text corpus to identify the tokens appearing in the corpus and then analyze them. The frequency of each token in the training text corpus of the target minority language is counted to obtain frequency statistics results.

4. The method for constructing speech recognition for minority languages ​​based on whispertoken according to claim 1, characterized in that, The input speech signal undergoes F-bank feature extraction. The extracted F-bank features are preprocessed and then fed into the Encoder for forward computation to generate high-level semantic features, including: Fbank features are extracted from the input speech signal to obtain the spectral feature representation of the speech signal. The extracted Fbank features are then preprocessed. The preprocessed Fbank features are input into the Encoder, which converts the input Fbank features into a high-level semantic feature representation through forward computation.

5. A speech recognition system for minority languages ​​based on whispertoken, the system implementing the method as described in any one of claims 1 to 4, characterized in that, include: The acquisition module is used to extract all tokens related to the target minority language from Whispertokenizer to form an initial candidate set; the tokens in the initial candidate set are matched and analyzed with the collected training text corpus of the target minority language, and the frequency of token occurrence in the corpus is counted. The module is used to filter high-frequency tokens and supplement low-frequency tokens based on frequency statistics to build a dynamic vocabulary. A speech recognition model is constructed, which includes an encoder, a decoder, a CTC decoder, and a CTC loss function. The model is trained using a dynamic vocabulary. The processing module is used to extract F-bank features from the input speech signal, preprocess the extracted F-bank features, and input them into the Encoder for forward computation to generate high-level semantic features. The high-level semantic features are processed by CTCDecoder, and multiple candidate results are generated through the BeamSearch strategy. The AttentionDecoder is used to re-score these candidate results, and the result with the highest score is selected as the preliminary recognition result. The preliminary recognition result is then post-processed to obtain the final text recognition result.

6. A computing device, characterized in that, include: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed by a processor, implements the method as described in any one of claims 1 to 4.

Images